[{"content":"Introduction I intend to write a blog about knowledge in Mathematics, Programming, AI, Lean4. However, it\u0026rsquo;s difficult to write such blog better than an article with the same topic in a textbook. So, I change my intention into write a personal blog whose contents will focus more on my personal thoughts, philosophy and maybe few mathematical details only if it has a meaningful story behind.\nBests, Minh Hai\n","permalink":"https://IntArchive.github.io/posts/brief_introduction/","summary":"\u003ch2 id=\"introduction\"\u003eIntroduction\u003c/h2\u003e\n\u003cp\u003eI intend to write a blog about knowledge in Mathematics, Programming, AI, Lean4. However, it\u0026rsquo;s difficult to write such blog better than an article with the same topic in a textbook. So, I change my intention into write a personal blog whose contents will focus more on my personal thoughts, philosophy and maybe few mathematical details only if it has a meaningful story behind.\u003c/p\u003e\n\u003cp\u003eBests,\nMinh Hai\u003c/p\u003e","title":"A Brief Introduction About My Blog"},{"content":"Introduction Suppose we observe pairs $(X_1, Y_1), \\ldots, (X_n, Y_n)$ where we want to estimate the regression function\n$$ m(x) = \\mathbb{E}[Y \\mid X = x]. $$The classical Nadaraya–Watson estimator is defined as\n$$ \\hat{m}_n(x) = \\frac{\\sum_{i=1}^n K_h(x - X_i)\\, Y_i}{\\sum_{i=1}^n K_h(x - X_i)}, $$where $K_h(u) = h^{-1} K(u/h)$ is a kernel with bandwidth $h \u003e 0$.\nThe Recursive Version In my thesis, I studied a recursive variant suited for streaming data. Define the sequence of estimates:\n$$ \\hat{m}_{n+1}(x) = \\hat{m}_n(x) + \\gamma_{n+1}\\, K_{h_{n+1}}(x - X_{n+1}) \\bigl[Y_{n+1} - \\hat{m}_n(x)\\bigr], $$where $\\{\\gamma_n\\}$ is a step-size sequence satisfying the Robbins–Monro conditions:\n$$ \\sum_{n=1}^\\infty \\gamma_n = \\infty, \\qquad \\sum_{n=1}^\\infty \\gamma_n^2 \u003c \\infty. $$Convergence Result Under mild regularity conditions on the kernel $K$ and the density $f$ of $X$, one can show:\n$$ \\hat{m}_n(x) \\xrightarrow{a.s.} m(x) \\quad \\text{as } n \\to \\infty. $$Moreover, the estimator satisfies an asymptotic normality result:\n$$ \\sqrt{n h_n}\\,\\bigl(\\hat{m}_n(x) - m(x)\\bigr) \\xrightarrow{d} \\mathcal{N}\\!\\left(0,\\; \\frac{\\sigma^2(x)\\,\\|K\\|_2^2}{f(x)}\\right), $$where $\\sigma^2(x) = \\mathrm{Var}(Y \\mid X = x)$ and $\\|K\\|_2^2 = \\int K^2(u)\\,du$.\nR Implementation Here\u0026rsquo;s the core vectorized implementation I used to simulate ECG waveforms:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 #\u0026#39; Recursive Nadaraya-Watson Estimator #\u0026#39; #\u0026#39; @param x Grid of evaluation points #\u0026#39; @param X Observed covariates (n x 1) #\u0026#39; @param Y Observed responses (n x 1) #\u0026#39; @param h_fn Bandwidth function h(n) -\u0026gt; numeric #\u0026#39; @param gamma_fn Step-size function gamma(n) -\u0026gt; numeric #\u0026#39; @return Matrix of estimates: nrow = length(x), ncol = n recursive_nw \u0026lt;- function(x, X, Y, h_fn, gamma_fn) { n \u0026lt;- length(X) m \u0026lt;- length(x) est \u0026lt;- matrix(0, nrow = m, ncol = n) # Gaussian kernel (vectorized over x at each step) K \u0026lt;- function(u) dnorm(u) for (i in seq_len(n)) { h \u0026lt;- h_fn(i) gamma \u0026lt;- gamma_fn(i) k_vec \u0026lt;- K((x - X[i]) / h) / h # m-length vector if (i == 1L) { est[, i] \u0026lt;- k_vec * Y[i] / (k_vec + 1e-12) } else { prev \u0026lt;- est[, i - 1L] est[, i] \u0026lt;- prev + gamma * k_vec * (Y[i] - prev) } } est } # Example: h(n) = n^{-1/5}, gamma(n) = 1/n result \u0026lt;- recursive_nw( x = seq(-pi, pi, length.out = 200), X = rnorm(1000), Y = sin(rnorm(1000)) + rnorm(1000, sd = 0.2), h_fn = function(n) n^(-1/5), gamma_fn = function(n) 1 / n ) The vectorization over the grid x at each step avoids the inner loop, giving roughly a 30× speedup compared to a naïve double-loop implementation.\nKey Takeaways The recursive estimator updates in $\\mathcal{O}(m)$ per observation (vs $\\mathcal{O}(nm)$ for batch NW). Choosing $h_n \\sim n^{-1/(4+d)}$ is optimal for $d$-dimensional $X$. The step-size $\\gamma_n = c/n$ satisfies Robbins–Monro and gives $\\sqrt{n}$-rate convergence under standard assumptions. References Fraysse, P. (2014). Recursive Estimation in a Class of Models of Deformation. Journal of Statistical Planning and Inference. Nadaraya, E. A. (1964). On estimating regression. Theory of Probability and Its Applications, 9(1), 141–142. Watson, G. S. (1964). Smooth regression analysis. Sankhyā, Series A, 26, 359–372. ","permalink":"https://IntArchive.github.io/posts/nadaraya-watson-recursive-estimation/","summary":"\u003ch2 id=\"introduction\"\u003eIntroduction\u003c/h2\u003e\n\u003cp\u003eSuppose we observe pairs $(X_1, Y_1), \\ldots, (X_n, Y_n)$ where we want to estimate\nthe \u003cstrong\u003eregression function\u003c/strong\u003e\u003c/p\u003e\n$$\nm(x) = \\mathbb{E}[Y \\mid X = x].\n$$\u003cp\u003eThe classical \u003cstrong\u003eNadaraya–Watson estimator\u003c/strong\u003e is defined as\u003c/p\u003e\n$$\n\\hat{m}_n(x) = \\frac{\\sum_{i=1}^n K_h(x - X_i)\\, Y_i}{\\sum_{i=1}^n K_h(x - X_i)},\n$$\u003cp\u003ewhere $K_h(u) = h^{-1} K(u/h)$ is a kernel with bandwidth $h \u003e 0$.\u003c/p\u003e\n\u003ch2 id=\"the-recursive-version\"\u003eThe Recursive Version\u003c/h2\u003e\n\u003cp\u003eIn my thesis, I studied a \u003cstrong\u003erecursive\u003c/strong\u003e variant suited for streaming data.\nDefine the sequence of estimates:\u003c/p\u003e","title":"Nadaraya–Watson Recursive Estimation: Theory and Implementation"},{"content":"Inline Math Einstein\u0026rsquo;s famous equation: $E = mc^2$\nDisplay Math The Gaussian integral:\n$$ \\int_{-\\infty}^{\\infty} e^{-x^2} dx = \\sqrt{\\pi} $$Maxwell\u0026rsquo;s equations in differential form:\n$$ \\nabla \\cdot \\mathbf{E} = \\frac{\\rho}{\\varepsilon_0} $$Code Snippet 1 2 3 4 5 6 7 8 def fibonacci(n: int) -\u0026gt; int: \u0026#34;\u0026#34;\u0026#34;Return the nth Fibonacci number.\u0026#34;\u0026#34;\u0026#34; if n \u0026lt;= 1: return n a, b = 0, 1 for _ in range(2, n + 1): a, b = b, a + b return b ","permalink":"https://IntArchive.github.io/posts/my-first-post/","summary":"\u003ch2 id=\"inline-math\"\u003eInline Math\u003c/h2\u003e\n\u003cp\u003eEinstein\u0026rsquo;s famous equation: $E = mc^2$\u003c/p\u003e\n\u003ch2 id=\"display-math\"\u003eDisplay Math\u003c/h2\u003e\n\u003cp\u003eThe Gaussian integral:\u003c/p\u003e\n$$\n\\int_{-\\infty}^{\\infty} e^{-x^2} dx = \\sqrt{\\pi}\n$$\u003cp\u003eMaxwell\u0026rsquo;s equations in differential form:\u003c/p\u003e\n$$\n\\nabla \\cdot \\mathbf{E} = \\frac{\\rho}{\\varepsilon_0}\n$$\u003ch2 id=\"code-snippet\"\u003eCode Snippet\u003c/h2\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cdiv class=\"chroma\"\u003e\n\u003ctable class=\"lntable\"\u003e\u003ctr\u003e\u003ctd class=\"lntd\"\u003e\n\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode\u003e\u003cspan class=\"lnt\"\u003e1\n\u003c/span\u003e\u003cspan class=\"lnt\"\u003e2\n\u003c/span\u003e\u003cspan class=\"lnt\"\u003e3\n\u003c/span\u003e\u003cspan class=\"lnt\"\u003e4\n\u003c/span\u003e\u003cspan class=\"lnt\"\u003e5\n\u003c/span\u003e\u003cspan class=\"lnt\"\u003e6\n\u003c/span\u003e\u003cspan class=\"lnt\"\u003e7\n\u003c/span\u003e\u003cspan class=\"lnt\"\u003e8\n\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/td\u003e\n\u003ctd class=\"lntd\"\u003e\n\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-python\" data-lang=\"python\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"k\"\u003edef\u003c/span\u003e \u003cspan class=\"nf\"\u003efibonacci\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003en\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e \u003cspan class=\"nb\"\u003eint\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e \u003cspan class=\"o\"\u003e-\u0026gt;\u003c/span\u003e \u003cspan class=\"nb\"\u003eint\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"s2\"\u003e\u0026#34;\u0026#34;\u0026#34;Return the nth Fibonacci number.\u0026#34;\u0026#34;\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"k\"\u003eif\u003c/span\u003e \u003cspan class=\"n\"\u003en\u003c/span\u003e \u003cspan class=\"o\"\u003e\u0026lt;=\u003c/span\u003e \u003cspan class=\"mi\"\u003e1\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"k\"\u003ereturn\u003c/span\u003e \u003cspan class=\"n\"\u003en\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"n\"\u003ea\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e \u003cspan class=\"n\"\u003eb\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"mi\"\u003e0\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e \u003cspan class=\"mi\"\u003e1\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"k\"\u003efor\u003c/span\u003e \u003cspan class=\"n\"\u003e_\u003c/span\u003e \u003cspan class=\"ow\"\u003ein\u003c/span\u003e \u003cspan class=\"nb\"\u003erange\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"mi\"\u003e2\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e \u003cspan class=\"n\"\u003en\u003c/span\u003e \u003cspan class=\"o\"\u003e+\u003c/span\u003e \u003cspan class=\"mi\"\u003e1\u003c/span\u003e\u003cspan class=\"p\"\u003e):\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"n\"\u003ea\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e \u003cspan class=\"n\"\u003eb\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"n\"\u003eb\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e \u003cspan class=\"n\"\u003ea\u003c/span\u003e \u003cspan class=\"o\"\u003e+\u003c/span\u003e \u003cspan class=\"n\"\u003eb\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"k\"\u003ereturn\u003c/span\u003e \u003cspan class=\"n\"\u003eb\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/td\u003e\u003c/tr\u003e\u003c/table\u003e\n\u003c/div\u003e\n\u003c/div\u003e","title":"Beautiful Math and Code"}]