Site-wide feed

A Scrapbook of Neural Network Lores

Lightly curated list of stories, anecdotes, and other various bits from neural network research.
Jan 18, 2024
Yuxi Liu

A brutalist representation of the particle-wave duality of the Hamiltonian--Jacobi equation. The stacked platforms represent the Hamilton principal function according to the field-theoretic wave interpretation, and the steel rebars are the characteristic rays, with balls rolling along them representing the particle interpretation. Made in Ideogram V1 with the following prompt: 'A brutalist terraced mountain made of concrete and steel rebar. There are ripples parallel to the ground. The steel rebars go downhill, and many marbles are rolling down along the steel rebars. White background. High contrast, grayscale, black-and-white minimalistic, in the flat style of vector svg art, conceptual art'

Analytical Mechanics

What every graduate student should know about analytical mechanics, delivered with economic style and many illustrations. Particular focus on particle-wave duality and old quantum theory. Prerequisites: multivariate calculus and mathematical maturity.
May 4, 2024
Yuxi Liu

 

Characteristic time in finance

Application of a simple concept from physics to personal finance.
May 1, 2025
Yuxi Liu

A brutalist sculpture representing classical thermodynamics/neoclassical economics: nonlinear (curved surface) constraint (steel cables tying it down) optimization (the surface struggling to fly towards the sun, touching the solar rays at the closest points). Made in Ideogram V1, with prompt 'A stunning conceptual artwork featuring a white sun casting intense white rays of sunlight, piercing through a curved hyperboloid structure made of concrete. The structure is adorned with straight lines of tight steel cables, creating a strong geometric pattern. The white background and high contrast accentuate the minimalistic vector art style. The piece is reminiscent of Chiaroscuro painting in the style of Piranesi, with a monochrome palette that enhances the drama and depth of the scene., illustration'.

Classical Thermodynamics and Economics

How to think like a classical thermodynamic-economist, delivered with many illustrations and some sci-fi metaphors. Particular emphasis on what traditional pedagogy gets wrong. Prerequisites: multivariate calculus and mathematical maturity.
May 9, 2024
Yuxi Liu

A grid of electromechanical spider-like robots on a concrete surface, with small cracks and many rounded-headed copper spires like tallgrass. It might be called poetically 'a cybernetic meadow'. Realistic sci-fi illustration.

Cybernetic Artificial Intelligence

Machine learning and self-reproduction according to Norbert Wiener.
Dec 23, 2023
Yuxi Liu

A symbolic representation of the Cyc project. The vanishing point covered by mist represents the project that keeps growing without ever reaching completion, a destination that is a mirage. The rows of pyramids make up a larger pyramid, due to perspective. This creates the sense of a recursive pyramid, a hierarchy of hierarchies, representing the hierarchy of microtheories in Cyc's ontology. The pyramid is also a monument to the dead, of vast ambitions that failed to secure their design purposes. For Egyptian pyramids, the purpose was a good afterlife. For Cyc, the purpose was AGI. A one-point perspective image of a path towards infinity, banked by two rows of pyramids. The pyramids are stepped pyramids made of concrete. The path fades into a distant mist. The point of view is above the tops of the pyramids. Monochromatic photo. Created by GPT-4o.

Cyc

Obituary for the greatest monument to logical AGI
After 40 years, 30 million rules, 200 million dollars, 2000 person-years, and many promises, Cyc has failed to reach intellectual maturity, and may never will. Exacerbated by the secrecy and insularity of Cycorp, there remains no evidence of its general intelligence.
Apr 1, 2025
Yuxi Liu

Dwarkesh Patel dinner party

Random recollections from the hectic conversations at a dinner party organized by Dwarkesh Patel.
May 30, 2025
Yuxi Liu

Brutalist architecture representing a neural network. A vaguely hexagonal 3D array of concrete blocks connected by thin 'walkways', small thin steel 'lampposts' topped with balls, and balls rolling around.

Fermi Estimation for Neural Networks

The bitter lesson in bite-sized packets.
Dec 5, 2023
Yuxi Liu

 

Fictional ideas

Ideas for stories, fictions, etc.
Jun 11, 2024
Yuxi Liu

Grokking Modular Arithmetics

Toy model of grokking: tiny neural networks learn modular arithmetics after it has already reached full correctness on the training set.
Sep 1, 2023
Yuxi Liu

Hole Argument and Inverted Qualia

The hole argument in general relativity is formally analogous to the inverted qualia problem in philosophy. Like how spacetime points have no existence beyond gauge freedom, qualias have no existence beyond their geometric-functional roles, thus dissolving the hard problem of consciousness.
Jun 18, 2024
Yuxi Liu

A metaphor for real-space renormalization, where the building on the left is coarse-grained to the building on right. The facade of a wide brutalist architecture. On the left, the windows are small and the floors are close together. On the right, the windows are large and the floors are wide apart. High contrast, monochromatic, minimalistic, in the flat style of vector svg art., illustration, conceptual art.

How to do Renormalization

Survival guide for renormalization (RN) theory. How to do it in real space and frequency space. What it all means. Should have something for everyone, from mathematicians to physicists to outsiders, from serious series to pretty pictures. Intended audience: those with two years of undergrad mathematics.
Apr 11, 2024
Yuxi Liu

 

How to do field-theoretic calculations

I teach you how to do probability calculations and gigantic integrals in the ‘field-theoretic style’, done in exhaustive details. I aim for clarity, pointing out every pitfall that I have fallen into so that you don’t have to.
Apr 11, 2024
Yuxi Liu

A swarm of mechanical insects moving to the right. Moving along with the insects are arrows with shafts made of barbed wires. It invokes the feeling of swarm warfare. High contrast, monochromatic, minimalistic, in the style of vector svg art.

Information Warfare

A mosaic view of the info-battleground: information for anti-informative goals. Intentionally useless errors, diplomatic speech, Chinese censorship, spam campaign, stock market arbitrage, continental philosophy, etc.
Dec 16, 2023
Yuxi Liu

Mixture of Experts

How MoE works, its history, and what it is good for.
Jan 23, 2024
Yuxi Liu

Neural Scaling Laws by Data Manifold Dimensions

Neural networks scale they way they do, purely because of data.
Nov 1, 2023
Yuxi Liu

Notes for a talk on the history of neural networks

Notes for a talk on the history of neural networks.
Aug 1, 2024
Yuxi Liu

 

Notes on Using Linux

My quick reference for using Linux for doing things.
Dec 10, 2022
Yuxi Liu

 

Notes on Web Design

My quick reference for designing content for the Internet.
Dec 10, 2023
Yuxi Liu

Perfect diffusion is TC0 – Bad diffusion is Turing-complete

An application of computational complexity theory to diffusion language models. Unlikely to be useful for anything, but it is cute!
Apr 17, 2025
Yuxi Liu

Phase-space quantum mechanics

Notes on phase-space quantum mechanics.
Sep 1, 2024
Yuxi Liu

Philosophical Sketches

Short philosophical writings, often poking fun at philosophers.
Jun 16, 2024
Yuxi Liu

An abstract representation of the Turing test. On the left is a dark city of obelisks over a bright background, and on the right is the same thing, mirrored and inverted, but with subtle differences due to the randomness of the art generator. It represents the abstract idea of the Turing test: duality, subtle differences, same contours, and complex object made of simple parts. High contrast, monochromatic, minimalistic, in the style of vector svg art.

Predicting AGI by the Turing Test

Minimizing log-perplexity loss is equivalent to maximizing survival length in a Turing test. Assuming compute-loss scaling law, a scaled-up GPT that produces human-like science papers would cost ~200 years of global GDP.
Jan 20, 2024
Yuxi Liu

The cover of 'Project SOGO: Principles of Hypnotic Bionics'. It is a combination of 'Project PARA: Principles of Neurodynamics' and 'Shoggoth' from Lovecraft's cosmology. The idea is that the book is mythical like the Necronomicon (see Minsky's quote). In the style of a cold-war era RAND corporation report.

Reading Perceptrons

A long, hard stare into the math of Perceptrons, the mythical neural network killer.
Dec 21, 2023
Yuxi Liu

 

Research ideas

Ideas for research. Free for a good home.
Jun 11, 2024
Yuxi Liu

A girl facing away from the viewer, sitting in a datacenter, connected by wires to the server racks. In the middle-top is an ethereal tree-shaped network of light, representing the neural network of Lain. Cyberpunk-style anime art.

Serial Experiments Lain

Lain is the collective subconscious as a neural network.
Nov 1, 2022
Yuxi Liu

A bricolage representing modern statistical mechanics in all its multifaceted chaotic order. Containing numbers (information), particles in a box, Jupiter's great red eye, RNA hairpins, bacteria with wavy flagella. Made in Ideogram V1, with prompt 'An abstract conceptual illustration of a brutalist structure, with sharp angles and a minimalistic design, made of concrete, steel balls, and rebars. The structure appears to be made up of particles in a box, jamming together, creating bits and numbers in the realm of 0-1. The design also features an RNA hairpin, with eddies upon eddies upon eddies, evoking the image of Jupiter's red spot. The random walk of swimming bacteria with wavy flagella is depicted, adding to the complexity of the design. The illustration is presented on a white background, with high contrast and a monochromatic color scheme, reminiscent of a flat style vector SVG art., illustration, conceptual art'.

Statistical Mechanics

How to think like a classical statistical-mechanic, with many examples from gas theory, biology, probability, and information theory. Prerequisites: thermodynamics, calculus, probability, and mathematical maturity.
Jul 4, 2024
Yuxi Liu

Made by GPT, with prompt: 'A shining red five-pointed star on the top left, sending out multiple light rays. It is in the similar style as socialist iconography. In the rest of the image, four candlestick charts overlapping and growing exponentially from the bottom left to the top right. No vignetting. Widest aspect ratio.'.

Structure and Interpretation of the Chinese Economy

My basic framework for understanding the modern Chinese economy and its growth. Includes brief histories of Chinese, Soviet, and Japanese economy. Framework applied to two case studies on the stock market and the housing market.
Jun 3, 2025
Yuxi Liu

That Survivorship Bias Plane

The exact backstory to that picture of an airplane with red dots on top of it.
May 2, 2025
Yuxi Liu

Stylized illustration of a computational graph, with gibberish text that resemble vector calculus, thick and thin lines representing the flow of numbers in the graph. In the center of the vaguely octagonal graph is a letter D, D for derivative. High contrast, monochromatic, minimalistic, in the style of vector svg art.

The Backstory of Backpropagation

Why backprop was resisted for 20 years: assumption of discretely spiking neurons, goal of synthesizing Boolean logic, fear of local optima, and bad luck. Werbos has the best claim for invention.
Dec 26, 2023
Yuxi Liu

Brutalist architecture representing a single perceptron. A hexagonal building in the center representing the unit itself, with multiple arc legs representing the input weights, and antennas projecting off the top representing the output. High contrast, monochromatic, minimalistic, in the style of vector svg art.

The Perceptron Controversy

Connectionism died in the 60s from technical limits to scaling, then resurrected in the 80s after backprop allowed scaling. The Minsky–Papert anti-scaling hypothesis explained, psychoanalyzed, and buried.
Jan 1, 2024
Yuxi Liu

The Racial Algorithmic Bias Controversy

The racial algorithmic bias controversy, as seen by a mathematician.
Jul 13, 2019
Yuxi Liu

A picture of iterating the same Lorentz boost for many times on a plus-sign with unit-length arms, creating the outlines of a hyperbola.

The Wigner Rotation in Special Relativity via Hyperbolic Geometry

The other special relativity paradox that you have never heard of.
May 1, 2018
Yuxi Liu

The cost of O’Neil habitation

To win (?) a twitter argument, we calculated the energetic cost of O’Neil habitat per person. Oddly, it is just affordable by modern standards. About 0.6 million dollars per person. Let it be known that the SF housing crisis is cheaper to solve by building O’Neil cylinders.
Apr 18, 2025
Yuxi Liu

Whalefall

Thoughts on the recent ‘DeepSeek shock’ of the stock market. The market is being even less efficient than usual, and much of the ‘professional’ commentators are idiots or bullshitters. I wrote this post to prevent my Gell-Mann amnesia.
Jan 29, 2025
Yuxi Liu
No matching items